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EXECUTIVE SUMMARY 


1.0 Introduction 


\ 


This summary serves as a stand-alone document, as well as part of this report. For that reason, 
the reader will find some duplication of verbiage and figures between the summary and the full 
report. 


2.0 JADS Overview 


The Joint Advanced Distributed Simulation (JADS) Joint Test and Evaluation (JT&E) was 
chartered by the deputy director, Test, Systems Engineering and Evaluation (Test and 
Evaluation), Office of the Under Secretary of Defense (Acquisition and Technology) in October 
1994 to investigate the utility of advanced distributed simulation (ADS) technologies for support 
of developmental test and evaluation (DT&E) and operational test and evaluation (OT&E). The 
program is Air Force led with Army and Navy participation. JADS Joint Test Force (JTF) 
manning currently includes 18 Air Force, 4 Air Force civilians, 12 Army, and 1 Navy civilian. 
Science Applications International Corporation and the Georgia Tech Research Institute provide 
contracted technical support. The program is currently scheduled to end in March 2000. 


The JADS JTF is directly investigating ADS applications in three slices of the test and evaluation 
(T&E) spectrum: the System Integration Test (SIT) which explored ADS support of air-to-air 
missile testing; the End-to-End (ETE) Test which is investigating ADS support for command, 
control, communications, computers, intelligence, surveillance, and reconnaissance (C4ISR) 
testing; and the Electronic Warfare (EW) Test which is exploring ADS support for EW testing. 
The JTF is also chartered to observe or participate at a modest level in ADS activities sponsored 
and conducted by other agencies in an effort to broaden conclusions developed in the three 
dedicated test areas. | 


Phase 2, the laboratory test, of the ETE Test is the subject of this summary report. 
3.0 ETE Test Overview 


The ETE Test is designed to evaluate the utility of ADS to support testing of C4ISR systems. 
The test uses the Joint Surveillance Target Attack Radar System (Joint STARS) as one 
component of a representative C4ISR system. The ETE Test also evaluates the capability of the 
JADS Test Control and Analysis Center (TCAC) to contro] a distributed test of this type and 
remotely monitor and analyze test results. 


The ETE Test consists of four phases. Phase 1 developed or modified the components needed to 
develop the ADS test environment. Phase 2 used the ADS test environment to evaluate the utility 
of ADS to support DT&E and early OT&E of a C4ISR system in a laboratory environment. 
Phase 3 transitions portions of the architecture to the E-8C aircraft, ensures that the components 
function properly, and checks that the synthetic environment interacts properly with the aircraft 


and actual light ground station module (LGSM). Phase 4 will evaluate the ability to perform test 
and evaluation of the E-8C and LGSM in a synthetically enhanced live test environment. 


4.0 Overview of ETE Test Phase 2 
4.1 Purpose 


Phase 2 determined the utility of ADS to support DT&E and early OT&E of a C4ISR system in a 
laboratory environment. Using the components developed in Phase 1, a distributed interactive 
simulation (DIS) network with appropriate C4ISR and weapon system nodes was developed to 
evaluate a representative sensor-to-shooter process. A virtual Army Tactical Missile System 
(ATACMS) battalion (Bn) was used as the engagement system. The test objectives were 


JADS Issue 1. What is the present utility of ADS, including DIS, for T&E? 


JADS Objective 1-1. Assess the validity of data from tests utilizing ADS, including DIS, 
during test execution. 


JADS Objective 1-2. Assess the benefits of using ADS, including DIS, n T&E. This test 
objective was broken down into subobjectives. (Subobjective 1-2-1 is not applicable to the 
ETE Test Phase 2.) 


JADS Subobjective 1-2-2. Assess ADS capability to support T&E planning and test 
rehearsal. 


JADS Subobjective 1-2-3. Assess ADS capability to support T&E shortfalls. 


JADS Issue 2. What are the critical constraints, concerns, and methodologies when using ADS 
for T&E? : 


JADS Objective 2-1. Assess the critical constraints and concerns in ADS performance for 
T&E. This objective was broken down into subobjectives. (Subobjective 2-1-1 is not 
applicable to the ETE Test Phase 2.) 


JADS Subobjective 2-1-2. Assess network and communications performance constraints 
and limitations. 


JADS Subobjective 2-1-3. Assess the impact of ADS reliability, availability, and 
maintainability on T&E. 


JADS Objective 2-2. Assess the critical constraints and concerns in ADS support systems 
for T&E. This objective was broken down into subobjectives. 


JADS Subobjective 2-2-1. Assess the critical constraints and concerns regarding ADS 
data management and analysis systems. 


JADS Subobjective 2-2-2. Assess the critical constraints and concerns regarding 
configuration management of ADS test assets. 


JADS Objective 2-3. Develop and assess methodologies associated with ADS for T&E. 
(Subobjective 2-3-1 is not applicable to the ETE Test Phase 2.) 


JADS Subobjective 2-3-2. Develop and assess methodologies associated with test 
execution and control for tests using ADS. 


4.2 Approach 


Figure ES-1 provides an overview of the ETE Test synthetic environment. 


TCAC 


Albuquerque, New Mexico 


AFATDS = Advanced Field Artillery Tactical Data System Janus = interactive, computer-based simulation of combat 
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Figure ES-1. ETE Test Phase 2 Synthetic Environment 


The ETE Test used the Janus 6.88D simulation to generate the entities representing the elements 
in the rear of a threat force. The U.S. Army Training and Doctrine Command Analysis Center 
(TRAC) at White Sands Missile Range (WSMR), New Mexico, provided the Janus scenario feed. 


The Test Control and Analysis Center, in Albuquerque, New Mexico, provided test control. 


The Joint STARS E-8C simulation, called the Virtual Surveillance Target Attack Radar System 
(VSTARS), represented the radar subsystem of the Joint STARS E-8C in a laboratory 
environment. It was composed of a distributed interactive simulation network interface unit 
(NIU), a radar processor simulator and integrator (RPSD) that contained the two real-time radar 
simulations with necessary databases, and various simulations of E-8C processes. Figure ES-2 


provides more information on the VSTARS architecture. VSTARS was operated at the Northrop 
Grumman Surveillance and Battle Management Systems facility in Melbourne, Florida. 
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Figure ES-2. VSTARS Architecture 


The LGSM and target analysis cell (TAC) were represented by Bravo Company, 303d Mulitary 
Intelligence Battalion. Fire support, in the form of the Advanced Field Artillery Tactical Data 
System (AFATDS), was represented by soldiers from the 4th Infantry Division (Mechanized). 


Communications among these command, control, communications, computers and intelligence 
(C41) systems employed such doctrinally correct means as the CGS-100, a subsystem of the 
Compartmented All Source Analysis System (ASAS) Message Processing System (CAMPS), 
remote workstations (RWSs), and AFATDS message traffic. 


The Tactical Army Fire Support Model (TAFSM) simulation modeled the Army Tactical Missile 
System (ATACMS) battalion and sent the fire and detonate protocol data units (PDUs) to the 
Janus 6.88D simulation. Janus then modeled the engagement results and reflected them in the 
synthetic environment. 


5.0 ETE Test Phase 2 Results 
5.1 Schedule 


The overall ETE Test schedule is presented in Figure ES-3. Phase 2 testing proceeded as 
scheduled. 
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Figure ES-3. ETE Test Schedule 


5.2 Fulfillment of Test Objectives 


All ETE Test Phase 2 objectives were met. The ETE Test team determined that ADS testing can 
be beneficial for test planning, rehearsal, and execution, and can result in valid data being 
collected. During Phase 2, they also identified critical constraints, concerns, and methodologies 
associated with using ADS for test and evaluation. Finally, the ETE Test team developed and 
assessed test control and data collection methodologies useful for ADS testing. 


6.0 Lessons Learned 
6.1 Technical 


Testers should carefully plan the development of the simulations and links comprising their ADS 
environment. During test execution, they must ensure that the time sources are synchronized and 
continuously monitor PDU traffic. The distributed nature of ADS testing necessitates special 
equipment for network check-out and verification and requires strict configuration control of 
analysis tools and collected data. 


6.2 Infrastructure and Process 


ADS test planning should be detailed enough to encompass key requirements at the earliest 
possible stages, yet flexible enough to accommodate unexpected situations during test execution. 
A conservative development approach is recommended -- accomplish risk reduction activities 
before each ADS test and let each ADS test build on the success of earlier experiments. 
Successful test execution requires effective internode communication, test and resource control, 
and data management procedures. 


7.0 Conclusions 
7.1 Utility 


An ADS environment can enhance C4ISR system DT&E and OT&E. In comparison with 
conventional tests, ADS allows testers to examine C4ISR systems under realistic conditions for 
longer periods of time, over far larger battlespaces, and at a much lower cost. This versatile 
technology can provide test environments that include large numbers of entities, entities operating 
under realistic but unsafe conditions, and joint and combined operations. ADS provides C4ISR 
system testers with greater flexibility in designing, executing, and analyzing their tests. During 
DT&E, ADS allows for more realistic compliance testing of C4ISR subsystems and efficient 
implementation of the test-fix-verify cycle for software development. 


ADS testing provides dramatic abilities to test C4ISR systems or components in that system. For 
instance, in the ETE Test ADS environment, a developmental test could be performed on the 
Joint STARS operations and control subsystem using VSTARS as a stimulus. With a similar 
configuration, an operational test could be accomplished on a LGSM. The ETE Test ADS 
environment also provides ample opportunities to install new components for various types of 
testing. Links to airborne weapon system simulators, complimentary sensor feeds or other 
command and control structures can be easily accomplished. The development of an ADS test 
environment during system development greatly improves opportunities for C4ISR system 
training after the completion of the test. The same infrastructure developed for testing can be 
modified and transitioned to a training environment resulting in program savings. This technology 
allows C4ISR system operators to confirm current tactics, try “what-if” scenarios and new tactics, 


test the interoperability and compatibility of their equipment, and gain useful experience in a 
realistic operating environment containing multiple assets. 


7.2 Technical 


The Phase 2 test required only a small part of the available bandwidth and exhibited a low PDU 
latency rate comparable with earlier tests. The ETE Test network was highly reliable during 
Phase 2 testing due largely to the ETE Test team’s extensive pretest risk reduction efforts. 


7.3 Infrastructure 


Compared to conventional testing, ADS testing reduces the need for large numbers of fielded 
personnel and vehicles. The ability to automatically collect and analyze test data also reduces the 
number of people required for setup, execution, and analysis. ADS test success relies on well- 
organized test control and data management procedures. Distributed testing requires 
sophisticated instrumentation, trained personnel to operate and maintain that equipment, and 
funds to support personnel and equipment at distant test nodes. 


1.0 Introduction 


1.1 Joint Advanced Distributed Simulation Overview 


The Jomt Advanced Distributed Simulation (JADS) Joint Test and Evaluation (JT&E) was 
chartered by the deputy director, Test, Systems Engineering and Evaluation (Test and 
Evaluation), Office of the Under Secretary of Defense (OSD) (Acquisition and Technology) in 
October 1994 to investigate the utility of advanced distributed simulation (ADS) technologies for 
support of developmental test and evaluation (DT&E) and operational test and evaluation 
(OT&E). The program is Air Force led with Army and Navy participation. JADS Joint Test 
Force (JTF) manning currently includes 18 Air Force military, 4 Am Force civilians, 12 Army, 
and 1 Navy civilian. Science Applications International Corporation and the Georgia Tech 
Research Institute provide contracted technical support. The program is currently scheduled to 
end in March 2000. | 


The JADS JT&E charter focuses on three issues: what is the present utility of ADS, including 
distributed interactive simulation (DIS), for test and evaluation (T&E); what are the critical 
constraints, concerns, and methodologies when using ADS for T&E; and what are the 
requirements that must be introduced into ADS systems if they are to support a more complete 
T&E capability in the future. From these issues, objectives and measures have been developed to 
guide the evaluation. 


The JADS JTF is directly investigating ADS applications in three slices of the T&E spectrum: the 
System Integration Test (SIT) which explores ADS support of air-to-air missile testing; the End- 
to-End (ETE) Test which investigates ADS support for command, control, communications, 
computers, and intelligence, surveillance and reconnaissance (C4ISR) testing; and the Electronic 
Warfare (EW) Test which explores ADS support for EW testing. Each test will apply the JADS 
objectives and measures as appropriate to conduct their evaluation. The JTF is also chartered to 
observe or participate at a modest level in ADS activities sponsored and conducted by other 
agencies in an effort to broaden conclusions developed in the three dedicated test areas. 


The JADS ETE Test is the subject of this report and is described in the next section; the 
following is a brief synopsis of the SIT and EW Test. 


The SIT evaluated the utility of using ADS to support cost-effective testing of an integrated 
missile weapon/launch aircraft system in an operationally realistic scenario. The SIT also 
evaluated the capability of the JADS Test Control and Analysis Center (TCAC) to control a 
distributed test of this type and to remotely monitor and analyze test results. The SIT consisted 
of two phases each of which culminated in three flight missions. The missions simulated a single 
shooter aircraft launching an air-to-air missile against a single target aircraft. In the Linked 
Simulators Phase (LSP), the shooter, target, and missile were all represented by simulators. In 
the Live Fly Phase (LEP), the shooter and target were represented by live aircraft and the missile 
by a simulator. 


The EW Test will evaluate the utility of ADS in a distributed EW environment. The first phase 
was open air testing to develop a performance baseline for two subsequent test phases. The first 
distributed test phase employed a linked architecture using Department of Defense’s (DoD) high 
level architecture (HLA) which included a digital simulation model of the ALQ-131 self- 
protection jammer, threat simulation facilities, and constructive models which support replication 
of the open air environment. In the second phase, an installed systems test facility was substituted 
for the digital model. In both distributed test architectures, system performance data will be 
compared with live fly data for verification and validation (V&V). 


1.2 Test Overview 


The ETE Test is designed to evaluate the utility of ADS to support testing of C4ISR systems. It 
will conduct its T&E utility evaluation in an ADS-enhanced test environment, using the Joint 
Surveillance Target Attack Radar System (Joint STARS) as one component of a representative 
C4ISR environment. The ETE Test will also evaluate the capability of the JADS TCAC to 
control a distributed test of this type and to remotely monitor and analyze test results 


The ETE Test is using distributed simulation to assemble an enhanced environment for testing 
C4ISR systems. The intent is to provide a complete, robust set of interfaces from sensor to 
weapon system, including the additional intermediate nodes that would be found in a tactical 
engagement. The test will trace a thread of the complete battlefield process from target detection 
to target assignment and engagement at corps level using ADS. It will allow the tester to evaluate 
the thread as a whole or the contribution of any of the parts individually and to evaluate what 
effects an operationally realistic environment has on the system under test. 


The ETE Test is designed to add additional entities in a seamless manner to the battlefield seen by 
Joint STARS. In addition, adding some of the complementary suite of other command, control, 
communications, computers and intelligence (C4I) and weapon systems with which Joint STARS 
would interact will enable the test team to evaluate the utility of an ADS-enhanced test 
environment. 


The test concept (Figure 1) is to use ADS to supplement the operational environment experienced 
by the E-8C and light ground station module (LGSM) operators. By mixing available live targets 
with targets generated by a constructive model, a battle array approximating the major systems 
present in a notional corps area of interest can be presented. By constructing a network with 
nodes representing appropriate C4I and weapon systems, a more robust cross section of players is 
available for interaction with the E-8C and LGSM operators. 
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VSTARS = Virtual Surveillance Target Attack Radar System 


Figure 1. ETE Test Conceptual Model 


Several components were required to create the ADS-enhanced operational environment used in 
the ETE Test. In addition to Joint STARS, the ETE Test required a validated simulation capable 
of generating entities representing the rear elements of a threat force. As discussed in Section 
1.3.1, the ETE Test team selected the Janus simulation for this requirement. Also, simulations of 
the Joint STARS moving target indicator (MTI) radar and synthetic aperture radar (SAR) were 
needed to insert the simulated entities into the radar stream aboard the E-8C while it was flying a 
live mission. Other capabilities used to support the test include simulations or subsets of the 
Army’s artillery command and control process and a simulation of the Army Tactical Missile 
System (ATACMS). Communications among these simulations are accomplished using such 
doctrinally correct means as the CGS-100, a subsystem of the Compartmented All Source 
Analysis System (ASAS) Message Processing System (CAMPS), remote workstations (RWSs), 
and Advanced Field Artillery Tactical Data System (AFATDS) message traffic. 


The ETE Test consists of four phases. Phase 1 developed or modified the components that 
allowed the mix of live and simulated targets at an E-8C operator’s console and LGSM operator’s 
console. Phase 2 evaluated the utility of ADS to support DT&E and early OT&E of a C4ISR 
system in a laboratory environment. Phase 3 transitions portions of the architecture to the E-8C 
aircraft, ensures that the components function properly, and checks that the synthetic environment 
properly interacts with the aircraft and the actual LGSM. Phase 4 will evaluate the ability to 
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perform test and evaluation of the E-8C and LGSM in a synthetically enhanced operational 
environment using typical operators. 


1.3 Phase 1 Overview 


This section summarizes Phase 1 test activities which were completed in February 1998. During 
Phase 1, software and hardware needed to establish the ETE Test ADS environment were 
developed, modified, and integrated. In addition, Phases 2 through 4 were planned. 


The ETE Test ADS environment components developed during Phase 1 included a constructive 
simulation to provide virtual targets, an E-8C simulation called the Virtual Surveillance Target 
Attack Radar System (VSTARS), an interface to allow surveillance control data link (SCDL) 
traffic from VSTARS to be displayed in the LGSM, and an ADS network suitable for integration 
and testing. 


More detailed information on Phase 1 can be found in the End-to-End Interim Report, Phase 1, 
August 1998, available at http://www.JADS.abg.com. (After 1 March 2000 refer requests to HQ 
AFOTEC/HO, 8500 Gibson Blvd SE, Kirtland Air Force Base, New Mexico 87117-5558, or 
SAIC Technical Library, 2001 North Beauregard St. Suite 80, Alexandria, Virginia 22311.) 


1.3.1 Phase 1 Approach 


The JADS ETE Test team developed requirements for a constructive simulation and then 
evaluated available simulations against these requirements. The Janus simulation, developed and 
managed by the U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
(TRAC), White Sands Missile Range (WSMR), New Mexico, was selected as the simulation best 
able to be modified to meet JADS’ requirements. TRAC-WSMR expanded the Janus scenario 
driver into Janus 6.88D, a constructive simulation capable of supporting up to 10,000 individual 
entities with a distributed interactive simulation (DIS) interface to the ETE Test environment. 


The JADS ETE Test team investigated existing simulations of Jomt STARS and determined that 
none of them met the needed fidelity requirements. Northrop Grumman, the developer of the 
E-8C, created a laboratory emulation of the E-8C radar subsystem and the capability to integrate 
the E-8C into a synthetic environment. The VSTARS is a laboratory emulation of the E-8C radar 
subsystem and other aircraft components which can receive synthetic targets from a DIS network 
and provide the stimulus to display these targets on the Advanced Technology Work Station 
(ATWS) or LGSM. The radar processor simulator and integrator (RPSI) and the air network 
interface unit (ANIU) are parts of the VSTARS which are installed on the aircraft. The RPSI 
receives radar service requests (RSRs) from either an operator workstation (OWS) or a ground 
station module replica (GSMR) and provides radar target reports (moving target indicator and 
synthetic aperture radar to the OWS and GSMR. 


Phase 1 also included the development of a near real-time simulation of the E-8C synthetic 


aperture radar. The JADS ETE Test team, through the Advanced Research Projects Agency 
(ARPA) War Breaker project, conducted a trade study of various existing simulations. The 
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XPATCHES simulation, developed by Wright Laboratory (Dayton, Ohio) and Loral Defense 
Systems (Goodyear, Arizona), was selected as the best starting point for the E-8C SAR 
simulation. Lockheed Martin Tactical Defense Systems, Goodyear, Arizona, developed a SAR 
simulation emulating the Joint STARS SAR operation. This simulation system, referred to as the 
Advanced Radar Imaging Emulation System (ARIES), is operationally embedded into Northrop 
Grumman’s radar processor simulation and integrator. 


The normal connection between the E-8C and its associated LGSM is through a line-of-sight data 
link called the surveillance control data link (SCDL). After considerable investigation, the JADS 
ETE Test team determined that this link could not be easily transmitted via commercial 
communications lines. Based on discussions among the team, Northrop Grumman, and Motorola, 
the JADS ETE Test team decided to develop interfaces which transferred the normal message 
traffic between the E-8C and LGSM rather than attempting to directly transfer the SCDL 
messages. Northrop Grumman and Motorola developed an interface control document (ICD) 
which defined this message traffic. Northrop Grumman included the capability in VSTARS to 
capture these messages and divert them to an Ethernet; Motorola developed an interface unit 
between the LGSM and an Ethernet. This interface unit links the Ethernet with the internal 1553 
databus of the LGSM. Additionally, the interface unit simulates the operation of the ground data 
terminal requiring the LGSM operator to perform the normal linking process prior to receiving 
the message traffic from VSTARS. 


The Phase 1 network initially connected TRAC-WSMR with the JADS TCAC in Albuquerque, 
New Mexico, and was then extended from the TCAC to the Northrop Grumman laboratory 
facilities in Melbourne, Florida. Late in Phase 1, this network grew to include links from the 
TCAC to Fort Hood, Texas, and Fort Sill, Oklahoma, and a link between Northrop Grumman and 
Fort Hood. 


1.3.2 Phase 1 Results 


Phase 1 identified constraints associated with ADS testing. One key constraint was the ability of 
the DoD infrastructure to support ADS test and evaluation. A measure of this constraint is found 
in the amount of development required to establish a synthetic environment with which to conduct 
testing. Phase 1 provided insight onto the development required to support a test of this type. 
Phase | also demonstrated the application of a systems engineering methodology to identify the 
requirements for ADS components, evaluated the availability of ADS components, and modified 
or developed the components to meet the requirements. 


Extensive testing was accomplished to establish and verify the network configuration. (See 
Figures 4 and 5 for the final configuration.) Different hardware and software configurations were 
tested. The use of user data protocol! (UDP) packets and their impact on the ETE Test 
environment was tested. It was determined that UDP imposed some restrictions on network 
speed. Using UDP the routers were only capable of a throughput of 400 packets per second. 
UDP is used among all nodes with the exception of the SCDL and the Advanced Field Artillery 
Tactical Data System (AFATDS) to AFATDS connection. 
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Data management and analysis methods were examined. The use of data loggers was examined 
and various loggers were tested. One of these was developed in support of Janus; the other was a 
product from the U.S. Army Simulation, Training, and Instrumentation Command (STRICOM). 
Neither the STRICOM or Janus logger provided a high enough time-stamp accuracy, kept 
processor loads minimized, or provided flexible enough playback accuracy for the data analysis 
needs of the ETE Test. This led to the development of the JADS logger and the JADS player. 


The JADS logger was designed to place a priority on time stamping of protocol data units 
(PDUs). As PDUs are received they are time stamped. When the processor has time it logs the 
PDUs. It also has a very minimal user interface that forgoes graphics in favor of processor 
availability. 


The JADS player is able to operate in two modes. Using the first mode, an operator specifies the 
playback speed multiplier that controls the speed of the PDU playback. The second mode 
specifies the rate at which the PDUs will be played back. The player can also start playback at 
any log time. 


Synthetic environment analysis data tools were developed to aid in the analysis of PDU test data. 
They were all incorporated into a single tool called the JADS toolbox. Some of the features 
include PDU analysis in near real time, predefined analyses and outputs, conversion of binary 
logfile data to text data, PDU replay, and conversion utilities. The JADS logger was used 
extensively for the analysis of test data. 


Data management techniques explored during Phase 1 will be expanded upon during the 
remainder of the ETE Test program. It was proven that data collection of large data sets and 
analysis of the data are possible and even facilitated by the ETE Test environment. 


Phase 1 concluded with the creation of a large ADS synthetic environment for C4ISR testing. All 
the simulators were developed and then connected to create the ETE Test C4ISR testing 
environment. The environment provided the capability to test multiple systems simultaneously 
and to determine impacts of one system upon another system. Phase 2 took the environment to 
the next step and actually used it to conduct a C4ISR test. 
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2.0 Phase 2 Overview 
2.1 Phase 2 Purpose 


Phase 2 of the ETE Test was designed to determine the utility of ADS to support DT&E and 
early OT&E of a C4ISR system in a laboratory-based environment. 


2.2 Phase 2 Approach 


In Phase 2, the E-8C.aircraft was represented by ground-based simulations. The Janus model 
simulated the threat battlefield entities. VSTARS emulated the radar subsystem of the E-8C 
aircraft and provided the inputs needed to drive target displays on the ATWS (simulating displays 
onboard the E-8C aircraft) and the LGSM. The resulting radar reports were provided to an 
actual target analysis cell (TAC) for analysis and target assignment. Fire support missions were 
tasked using actual AFATDS messages sent to the Tactical Army Fire Support Model (TAFSM) 
which simulated the launch, flyout, and detonation of an ATACMS missile. 


Figure 2 shows the organizational structure for reporting and coordination during Phase 2 of the 


ETE Test. 
DDSA 
JADS JTF 


D&SA Il Corps, Fort Hood 
TRAC-WSMR Battle Lab 
Fire Support 504 MI Bde 
410 
Joint STARS ΡΟ 93d Wing 303 MIBn 
303 MI Bn 
TEXCOM Lab PM Joint STARS PBC 
B Co 
Northrop Grumman Lockheed Martin 


Bde = brigade Bn = battalion Co = company 
D&SA = Depth and Simultaneous Attack DDSA = deputy director, system assessment ID = infantry division 
JPO = joint program office MI = Military Intelligence PM = program manager 


TEXCOM = U.S. Army Test and Experimentation Command 


Figure 2. ETE Test Organizational Structure 


During ETE testing, the roles and responsibilities of these organizations are as follows. 
DDSA 
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The deputy director, system assessment (DDSA) in Washington, District of Columbia: 


Oversees the JADS Joint Test and Evaluation (JT&E) 


e Approves JADS financial requirements 

e Approves the program test plan (PTP) 

e Oversees the analysis and reporting of test results 
JADS JTF 


The JADS JTF in Albuquerque, New Mexico: 


Conducts overall planning, execution, analysis, and reporting of the test 

Manages funding to accomplish the test 

Develops and evaluates JADS issues, objectives, measures, and related data elements 
Develops and integrates the components of the ETE Test ADS environment 
Establishes necessary communication links with test participants 

Operates the Test Control and Analysis Center during tests 

Works with other organizations in analyzing test data 

Reports interim and final results to OSD 


TRAC-WSMR 
TRAC-WSMR, New Mexico: 


Develops, tests, and documents Janus 6.88D (an expanded variant of Janus) for JADS 
Assists in integrating Janus 6.88D into the ETE Test ADS environment 

Assists in database conversions 

Assists in developing vignettes 

Assists in verification, validation, and accreditation (VV&A) activities 

Assists in ETE Test execution 


TEXCOM Lab 
The U.S. Army Test and Experimentation Command (TEXCOM) Lab at Fort Hood, Texas: 


e Assists in scenario and vignette development 
e Assists in ETE Test execution 
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D&SA Battle Lab 
The Depth and Simultaneous Attack (D&SA) Battle Lab at Fort Sill, Oklahoma: 


e Provides and operates the TAFSM and AFATDS 
e Assists in the integration of the ETE Test ADS environment 
e Assists in VV&A activities and ETE Test execution 


U.S. Army III Corps 
ΠῚ Corps Headquarters at Fort Hood, Texas: 


e B Company (Co), 303d Military Intelligence (MI) Battalion (Bn), 504th MI Brigade (Bde) 
supports the conduct of ETE Test Phase 2 and Phase 4 events with LGSM(s) and a target 
analysis cell (TAC) and assists in the integration of the ETE Test ADS environment 

e 504 MI Bde provides a test environment for the ETE Test Phases 2 and 4 

e Fire Support 4th Infantry Division (4 ID) provides an AFATDS and personnel to support the 
ETE Test Phases 2 and 4 


Joint STARS Joint Program Office (JPO) 


Joint STARS JPO, Hanscom Air Force Base (AFB), Massachusetts, provides access to the Joint 
STARS JTF and Northrop Grumman. 


The Joint STARS JTF of the Joint STARS JPO in Melbourne, Florida: 


e Supports conduct of testing in all phases 
e Analyzes Joint STARS test results and provides evaluations according to JADS objectives 
e Assists in VV&A activities 


Northrop Grumman Aerospace Corporation 
Northrop Grumman, Electronics and Systems Integration Division in Melbourne, Florida: 


Designed, developed and integrated the radar processor simulation and integrator (RPSI) 
Developed the Virtual Surveillance Target Attack Radar System (VSTARS) 

Conducts and assists in verification and validation (V&V) activities 

Assists in E-8C mission planning 

Operates VSTARS during ETE Test phases 


Contracting with Northrop Grumman is conducted through Rome Laboratory in New York. 
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93d Wing 
93d Wing at Robins AFB, Georgia: 
e Provided operators during Phase 2 of the ETE Test. 


2.3 Test Objectives 


The JADS issues, test objectives, and subobjectives for Phase 2 are described below. Each 
subobjective in turn encompassed one or more test measures. In Section 4 these issues, 
objectives, subobjectives, and test measures are discussed in terms of their intent, the associated 
data collection methodology, and operational test results. 


JADS Issue 1. What is the present utility of ADS, including DIS, for T&E? 


JADS Objective 1-1. Assess the validity of data from tests utilizing ADS, including DIS, 
during test execution. 


JADS Objective 1-2. Assess the benefits of using ADS, including DIS, in T&E. This test 
objective was broken down into subobjectives. (Subobjective 1-2-1 is not applicable to the 
ETE Test Phase 2.) 3 


JADS Subobjective 1-2-2. Assess ADS capability to support T&E planning and test 
rehearsal. 


JADS Subobjective 1-2-3. Assess ADS capability to support T&E shortfalls. 


JADS Issue 2. What are the critical constraints, concerns, and methodologies when using ADS 
for T&E? 


JADS Objective 2-1. Assess the critical constraints and concerns in ADS performance for 
T&E. This objective was broken down into subobjectives. (Subobjective 2-1-1 is not 


applicable to the ETE Test Phase 2.) 


JADS Subobjective 2-1-2. Assess network and communications performance constraints 
and limitations. 


JADS Subobjective 2-1-3. Assess the impact of ADS reliability, availability, and 
maintainability on T&E. 


JADS Objective 2-2. Assess the critical constraints and concerns in ADS support systems 
for T&E. This objective was broken down into subobjectives. 
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JADS Subobjective 2-2-1. Assess the critical constraints and concerns regarding ADS 
data management and analysis systems. 


JADS Subobjective 2-2-2. Assess the critical constraints and concerns regarding 
configuration management of ADS test assets. 


JADS Objective 2-3. Develop and assess methodologies associated with ADS for T&E. 
(Subobjective 2-3-1 is not applicable to the ETE Test Phase 2.) 


JADS Subobjective 2-3-2. Develop and assess methodologies associated with test 
execution and control for tests using ADS. 


2.4 Phase 2 Methodology 
2.4.1 Tactical Vignettes 


The tactical vignettes for the ETE Test activities are unclassified. The ETE Test team enhanced a 
TRADOC-approved, 54-hour corps battlefield simulation (CBS) scenario by replicating an Iraqi 
corps rear area of operations in Iraq. Table 1 describes the five tactical vignettes created in Janus 
6.88D; each vignette is six hours. The following targets were present in the 150 x 150 kilometer 
(km) Southwest Asia (SWA) terrain box: air defense artillery (ADA) sites, command and control 


sites, lines of communications (convoys), logistics bases, and concentrations of armor and artillery 
units. 


Table 1. Vignettes Used During ETE Testing 


Vignette | Description Number of 
Merete [Pei Net 
| 1 | Prehostilityphase | 9,807 
| 2 | Preemptive strikes | 9.757 


I 

2 

3 Hammurabi Division logistical operations 9.904 
4 

5 : 


ἘΝ Commitment of the Hammurabi Division 9.781 
General headquarters (GHQ) depots to 
corps and divisional logistical operations 9.950 


2.4.2 Test Configuration 


2.4.2.1 Phase 2 Synthetic Environment 


Several components were required to create the ADS-enhanced environment used in Phase 2. 
Figure 3 provides an overview of the Phase 2 synthetic environment. 
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The ETE Test used the Janus 6.88D simulation to generate the entities representing the elements 
in the rear of a threat force. Janus generated entity state PDUs (ESPDUs) for the threat force 
which were transmitted to the E-8C simulation via the Test Control and Analysis Center (TCAC). 
TRAC-WSMR provided the Janus scenario feed. 


The TCAC in Albuquerque, New Mexico, provided test control. The JADS Network and 
Engineering (N&E) team monitored the health of the ETE Test network and ensured that 
adequate data flowed in support of the test. 


TCAC 


Albuquerque, New Mexico 
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T-1 = digital carrier used to transmit a formatted digital signal at 1.544 megabits per second 


Figure 3. ETE Test Phase 2 Synthetic Environment 


The Joint STARS E-8C simulation, VSTARS, represents the radar subsystem of the Joint STARS 
E-8C in a laboratory environment. It 15 composed of a distributed interactive simulation network 
interface unit (NIU), a radar processor simulator and integrator (RPSI) that contains the two real- 
time radar simulations with necessary databases, and various simulations of E-8C processes. 
Figure 4 provides more information on the VSTARS architecture. VSTARS was operated at the 
Northrop Grumman Surveillance and Battle Management Systems facility in Melbourne, Florida. 


The TAC, fire support (provided by the AFATDS), and a LGSM were stationed at Fort Hood, 
Texas. | 


Communications among these C4I systems employed such doctrinally correct means as the CGS- 
100, a subsystem of the CAMPS, remote workstations, and AFATDS message traffic. The 
AFATDS messages were transmitted between the AFATDS located at Fort Hood and the 
AFATDS located at Fort Sill using actual tactical protocols, rather than DIS PDUs. Also, the 
SCDL messages were transmitted between VSTARS and the LGSM using a dedicated link, a 
special-purpose interface, and the actual tactical protocols. 
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Figure 4. VSTARS Architecture 


The TAFSM simulation at Fort Sill modeled the ATACMS battalion and sent the fire and 
detonate PDUs to the Janus 6.88D simulation. In turn, Janus modeled the engagement results and 
reflected the results in the synthetic environment. 
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2.4.2.2 Phase 2 Network 


Figure 5 provides a more detailed description of the ETE Test network. 
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Figure 5. ETE Test Phase 2 Network Diagram 


2.4.2.3 Test Control and Monitoring 


During Phase 2, ETE Test and Network and Engineering team members performed test control 
from the TCAC. Test control consisted of three major areas -- network monitoring, 
communications, and test procedures. 


The Network and Engineering team conducted network monitoring in the TCAC using hardware 
and software tools. The software consisted of commercial products and test-specific tools 
developed by JADS analyst/programmers. They used the following systems. 

Silicon Graphics, Inc. (SGI) Indy - JADS logger 

SGI Indy - time server 

SGI Indigo - NETVisualyzer™ 

SUN SPARC 5 - Spectrum™ 

Line printers 
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JADS analyst/programmers developed the JADS logger. This software recorded all PDU traffic 
at individual sites. All nodes, with the exception of Fort Hood, had a JADS logger installed. The 
logger recorded the receipt of the PDU and time stamped it using an accurate time source. These 
data were used to analyze PDU transmission performance over the network. 


JADS analyst/programmers also developed a time server which provided the accurate time source 
needed for the JADS logger. The time server was tied to a global positioning system (GPS) 
receiver located in the TCAC and provided time to all nodes with an accuracy of 100 
microseconds. The software also contained monitoring tools to track the time servers 
performance over an 8-hour test period. 


Cabletron Spectrum'™, NETVisualizer™, and line printers were used to provide network 
monitoring. Spectrum’ measured bandwidth utilization. This tool recorded the percentage of 
bandwidth used, as well as bandwidth loading on a network segment. NETVisualizer™ software 
displayed real-time bandwidth use in a rolling bar graph format for quick visual reference. The 
line printers provided a printout of network router status. Any failure or high bit error rate 
resulted in a printout showing the problem and identity of the offending router. 


Communication among the distributed ETE Test nodes was critical. The TCAC provided all 
needed communication systems. Dedicated phone circuits, residing on the T-1 lines, provided 
classified and unclassified service. The unclassified line allowed connectivity among the TCAC, 
Fort Hood, Fort Sill, and TRAC-WSMR. The classified line allowed connectivity among the 
TCAC, Fort Hood, and Northrop Grumman. These lines allowed the operator to select the 
desired site, lift the receiver, and connect directly to that site. The TCAC could select multiple 
sites for conference calls on these lines. In addition to these lines, the ETE Test team used an 
unclassified conference line to coordinate such test events as network checks and after-action 
debriefs. This line allowed up to ten participants to connect at one time. 


Test procedures were required to provide effective control of all test nodes during test events. 
The test procedures were in checklist format, which provided for standardization among the 
distributed nodes. The network checklist was most critical and was used to initialize the network 
before the test. Other checklists included those used to start up hardware and software at 
individual nodes, as well as the checklist used by the TCAC test controller to start and stop the 
overall test. 
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2.5 Planned Phase 2 Schedule 


Figure 6 provides a schedule of the top level tasks for Phase 2 of the ETE Test. 
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Figure 6. ETE Test Schedule 
2.6 Phase 2 Costs 


This report does not describe the costs of the Phase 2 ETE Test. Rather, the report on Phase 4 of 
the ETE Test will include a Work Breakdown Structure covering the costs of all four phases of 
the ETE Test. The Phase 4 ETE Test report will be published in summer, 1999. 
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3.0 Phase 2 Execution Results 


Four functionality and integration (FI) tests and one risk reduction test were conducted prior to 
Phase 2 operational testing. 


3.1 ETE Test Functionality and Integration (FI) Tests #1 and #2 


FI tests #1 and #2 took place in spring 1998. These two tests focused on establishing the 
individual nodes and links comprising the ETE Test ADS environment and did not involve any 
data collection. 


3.2 ETE Test FI Tests #3 and #4 and Risk Reduction Test 


For all three tests, the ETE Test network consisted of Janus at WSMR, VSTARS at Melbourne, 
TAFSM at Fort Sill and the LGSM at Fort Hood. 


- During each test entity state PDUs (ESPDUs) were broadcast from WSMR to Northrop 
Grumman via the TCAC. In addition, entity state, fire, detonate, transmit, and signal PDUs were 
broadcast from Fort Sill to WSMR via the TCAC. 


- The Spectrum™ network analysis tool measured bandwidth utilization on the classified links 
connecting TCAC to Northrop Grumman and Northrop Grumman to Fort Hood. 


- Automated data were collected using PDU loggers at the nodes. Taking advantage of the 
networked nature of the ADS environment, JADS analysts used the file transfer protocol (FTP) to 
send the data collected at the nodes to JADS. These data were then compressed and converted 
for analysis on JADS UNIX'-based analysis tools. Operational data were collected using log 
sheets at each node. 


- Draft test procedures were documented before each test. During the test readiness review 
before each test, the baseline configuration was verified. During each test, the test controller 
revised and augmented all procedures used to control the test configuration, environment data, 
and ADS network. 


3.2.1 FI Test #3 


FI test #3 took place on 26-28 May 1998. 


- Janus vignettes 1 and 3 were used to simulate entities and movements for a threat corps rear 
area of operations. Vignette 1 added 9,897 entities to test execution; vignette 2 added 9,757. 


- Testing for most of day one was scrubbed because of a VSTARS malfunction. None of the 
nodes experienced any problems on days two or three. 
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- The network was unavailable for testing for 38 minutes out of a scheduled 20 hours and 33 
minutes. 


- For the TCAC-Northrop Grumman classified link the packet rate averaged 17 packets/second, 
network load (bandwidth utilized) during active test execution averaged 1%, and peak load never 
exceeded 30% of link capacity during the three days of testing. Packet rate and network load did 
increase significantly over the TCAC-Grumman link immediately following active testing each day 
because PDU logger data files were being imported from Northrop Grumman to the TCAC. 


- For the three-day test, total PDU losses and latency were 


Node A Node B PDUs Sent PDUs PDUs Lost ax/Mean 
Received Total/% gota (9) 


WSMR TCAC 934,971 872,297 | 62,674/6.70% | .026/.120/.029 
TCAC 872,297 804,567 | 67,730/7.76% | 0.031/.109/.034 
Fort Sill WSMR 2,543 2,513 30/1.12% | -- | 


3.2.2 FI Test #4 


FI test #4 took place on 25-26 June 1998. 


- Janus vignettes 1 and 2 were used to simulate entities and movements for a threat corps rear 
area of operations. Vignettes 1 and 2 each added 9,766 entities to test execution. 


- Testing on 25 June was scrubbed for less than 1 hour because of the power spike which 
disrupted ETE testing at WSMR. Both the Fort Hood and Northrop Grumman nodes 
experienced problems on 26 June with either a router or the Ethernet interface causing the 
problem. Fort Sill did not experience any problems on either day. 


- The network was unavailable for 1 hour and 13 minutes out of a scheduled 12 hours and 55 
minutes. 


- Packet rate experienced for the classified links averaged around 20 packets/second. Network 
load (bandwidth utilized) during active test execution averaged 1%, and peak load never exceeded 
5% of link capacity during the two days of testing. Packet rate and network load did increase 
significantly over the TCAC-Grumman link immediately following active testing each day because 
PDU logger data files were being imported from Northrop Grumman to the TCAC. 


- Total PDU losses and latency were 


Node A Node B PDUs Sent PDUs PDUs Lost x/Mean 
Received Total/% gre (s) 


WSMR TCAC 497,721 473,569 | 24,152/4.85% | .017/.139/.034 
TCAC 473,569 457,915 | 15,654/3.31% | .028/.060/.034 
Fort Sill WSMR 46/6.87% | -- 
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3.2.3 Risk Reduction Test 


The risk reduction test took place on 7-10 and 13 July 1998. 


- Janus vignettes 1 and 3. were used to simulate entities and movements for a threat corps rear 
area of operations. Vignette 1 added 9,897 entities to test execution; vignette 3 added 9,904 
entities. 


- On 7 July, Fort Sill experienced a minor problem because of a trunk card outage. Testing was 
interrupted for approximately 1 hour on ὃ July because of problems with Janus. Janus froze up, 
was restarted, and then froze again, causing the test to stop prematurely for that day. Both Fort 
Hood and Northrop Grumman experienced problems on 9 July because of difficulties with a 
router/Ethernet interface. Northrop Grumman also experienced a minor problem on 13 July 
because of a trunk card outage at JADS. 


- The network was unavailable for testing for 2 hours and 40 minutes out of a scheduled 35 hours 
and 7 minutes. 


- For the TCAC-Northrop Grumman classified link, packet rate averaged 19 packets/second, 
network load (bandwidth utilized) during active test execution averaged 1%, and peak load never 
exceeded 4% of link capacity during the five days of testing. For the Northrop Grumman - Fort 
Hood classified link, packet rate averaged 34 packets/second, network load averaged 1%, and 
peak load never exceeded 5% of link capacity. Packet rate and network load did increase 
significantly over the TCAC-Grumman link immediately following active testing each day because 
PDU logger data files were being imported from Northrop Grumman to the TCAC. 


- Total PDU losses and latency were as follows: 


Node A Node B PDUs Sent PDUs PDUs Lost | Min/Max/Mean 
Received Total/% Latency (s) 
WSMR TCAC 1,667,371 1,667,283 88/.005% .007/.142/.041 


TCAC 1,667,283 1,604,069 | 63,214/3.79% | .031/.361/.036 
Fort Sill WSMR 3,386 3,385 1.03% | 


3.3 Operational Test 


The operational test portion of the Phase 2 test took place from 14 September through 7 October 
1998. The specific measures addressed during the test and the data collected in support of those 
measures are discussed in Section 4. 
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4.0 Analysis of Test Objectives 


During the operational test portion of Phase 2 of the ETE Test, JADS analysts collected 
information to address the issues, JADS objectives, and test measures as outlined in the JADS 
Program Test Plan (PTP) and the ETE Test Data Management and Analysis Plan (DMAP). Only 
those subobjectives and measures evaluated using Phase 2 results are discussed. 


4.1 JADS Issue 1. What is the present utility of ADS, including DIS, for T&E? 


This objective determines the extent to which ADS technology can support the T&E of current 
and future C4ISR systems. 


4.1.1 JADS Objective 1-1. Assess the validity of data from tests utilizing ADS, including 
DIS, during test execution. 


During Phase 2 of the ETE Test, the ETE Test team examined the validity of data from an ADS 
configuration incorporating the VSTARS simulation, Janus simulation, and other components into 
a C4ISR architecture. 


JADS Measure 1-1-0-1. Degree to which ADS provides valid system under test (SUT) data. 


JADS Measure 1-1-0-2. Percentage of ADS data which are valid (data supporting test 
measures which are timely, accurate, rena’ and otherwise faithfully represent real world 
systems data). 


These two test questions gauge the ability of an ADS environment to provide valid data for a 
C4ISR system under test. The first measure addresses the validity of the SUT output data which 
form the data elements for evaluating SUT measures. The second measure provides an 
assessment of the input data provided to the SUT by the ADS environment. 


These measures were primarily addressed through implementation of the Phase 2 V&V plan. 
Since JADS is a DoD-sponsored joint test, the basis for the Phase 2 V&V was the DIS nine-step 
VV&A process model and its accompanying Recommended Practice for Distributed Interactive 
Simulation -- Verification, Validation, and Accreditation (draft-21 May 1996). Figure 7 provides 
an overview of the VV&A process model. Note that this model has been extensively discussed 
with numerous members of the DIS modeling and simulation community, has been generally 
accepted by V&V practitioners, and is currently being balloted as a standard. 
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Figure 7. DIS Nine-Step VV&A Process Model 


ETE Test VV&A represented a tailoring and implementation of the nine-step process to a 
multiservice test of a major system, Joint STARS, augmented with ADS. The tailored ETE Test 
process model used is described in the Phase 2 V&V reports, Phase 2 Verification and Validation 
Results for the End-to-End Test and Verification and Validation Report for Phase 2 of the 
Virtual Surveillance Target Attack Radar System (VSTARS). (Available from JADS, 2050A 2nd 
Street SE, Kirtland Air Force Base, New Mexico, 87117-5522. After 1 March 2000 refer 
requests to HQ AFOTEC/HO, 8500 Gibson Blvd SE, Kirtland Air Force Base, New Mexico 
87117-5558, or SAIC Technical Library, 2001 North Beauregard St. Suite 80, Alexandria, 
Virginia 22311.) 


The results from implementing the ETE Test process model for the V&V of the Phase 2 ADS 
configuration are detailed in the Phase 2 V&V reports and are summarized as follows. 


— Verification of Janus 6.88D. The following were verified. 

-- Janus 6.88D was capable of simulating at least 5000 entities with at least twenty-five 
percent moving. 

-- Janus 6.88D operators were capable of fully interacting with a scenario while it was 
running. In particular, the Janus site representative interacted with the scenario after 
several ATACMS impacts by altering the behavior of the surviving entities within the 
immediate area of the impact. 

-- Janus 6.88D accepted and processed scenarios with a duration longer than eight hours. 

-- Janus 6.88D was capable of proceeding at a pace representative of near real time. 

-- Janus 6.88D was capable of utilizing scenarios conducted upon terrain representing a 
simulation area of at least 170 km by 170 km. 
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Verification of VSTARS. The following were verified. 

-- VSTARS received and integrated virtual data from the Phase 2 ADS environment. 

-- VSTARS operated in three modes: live only, mixed live and virtual, and virtual only using 
the standard Joint STARS MTI message format. 

-- The radar timeline was not impacted by the MTI simulation. 

-- VSTARS processed parameter data in the same format as Joint STARS. 

-- VSTARS displayed live SARs in live areas of interest and virtual SARs in both mixed and 
virtual areas using the standard Joint STARS SAR message format. 

-- VSTARS permitted all of the installed operator workstation software to function without 
abnormal fault messages occurring with minor exceptions. 


Verification of compliance standards (DIS step 2). It was verified that the PDUs emitted by 

each simulation adhered to the prescribed format. 

-- It was verified that Janus 6.88D issued DIS 2.0.4 entity state protocol data units 
(ESPDUs) that conformed in content and format with the DIS 2.0.4 standards as amended 
by JADS. JADS modified the ESPDU time-stamp format from time passed since the 
beginning of the current hour to milliseconds since the beginning of the vignette. This 
allowed testers to trace an ESPDU back to a discrete event that occurred within the Janus 
vignette.) 

-- Note that the AFATDS located at Fort Hood communicated directly with the AFATDS at 
Fort Sill using standard AFATDS message traffic instead of DIS PDUs. 


Verification of compatibility (DIS step 6). It was verified that the modeling and simulation 

(M&S) components exchanged data and interacted appropriately with each other; _ that 

individual components correctly used the common data (e.g., terrain, weather) to generated 

their portion of the synthetic environment, and that the overall implementation was adequate 
to address the exercise requirements. It was also verified that the network allowed that 
transfer of information between the components without corruption and that the individual 
components shared a common perspective of the virtual reality produced by the exercise. 

Specific considerations were as follows. 

-- It was verified that Janus 6.88D received each ESPDU, fire PDU, and detonate PDU 
issued to it by the network and took the appropriate action as dictated by the PDU. In 
particular, Janus correctly identified ATACMS launches initiated by Fort Sill which were 
outside the scenario terrain box. 

-- It was verified that TAFSM accepted artillery missions using AFATDS messages and that 
TAFSM issued a fire and detonate PDU whenever an ATACMS missile was fired and 
subsequently detonated. 


Validation of the ETE Test Phase 2 synthetic environment (SE) (DIS Step 7). It was ensured 
that the integrated simulations were adequate to satisfy the ETE Test requirements such that 
the behavior and performance of the SE mapped sufficiently to real-world counterparts for the 
specific ETE Test application. The conclusion that the ETE Test SE was a realistic 
representation of the real world was based on structured interviews with the VSTARS, 
LGSM, and TAC operator subject matter experts (SME) participating in the test. 
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-- It was specifically validated that Janus 6.88D represented vehicle behavior to the degree 
detectable by the Joint STARS, as judged by Joint STARS operator SMEs viewing vehicle 
movement presented by the Joint STARS operator workstation. Realistic vehicle behavior 
was achieved after correcting an anomaly observed during the ETE risk reduction test. 

--- Portions of convoys missed turns and wandered off into the desert during the entire 
scenario. The lost portion of the convoy would then jump back into formation after a 
period of time and resume normal movement. 

--- It was determined that when many vehicles were moving Janus was not sending 
change of state PDUs at a high enough rate. When vehicles are moving, ESPDUs 
must be sent for any change of state (starting, stopping, turning, or changing speed 
beyond preset limits) in addition to the normal heartbeat ESPDUs for stationary 
entities. When Janus did not send enough change of state ESPDUs, VSTARS 
displayed the vehicle motion in a straight line from the last update. 

The solution was to turn off the heartbeat prior to the beginning of entity movement so 

that Janus issued only change of state ESPDUs as the changes occurred. The rate at 

which Janus sent change of state PDUs was also dramatically increased from a 

maximum rate of 15 PDUs per second to 100 PDUs per second. This allowed 

VSTARS to more accurately display the convoys movement on their true path. 

-- For VSTARS validation, operators with Joint STARS experience performed a two-hour 
mission using VSTARS and were then asked to compare VSTARS performance with 
Joint STARS. 

--- As with any simulation, differences were noted, but test participants still characterized 
VSTARS as having the same capabilities and limitations as Joint STARS. They could 
not identify any VSTARS process or function that limited their ability to perform their 
mission/job or altered their approach to their mission/job. Results demonstrated a 
close correspondence between VSTARS and Joint STARS. 


Conclusion for JADS Measure 1-1-0-1. The Phase 2 ADS configuration produced valid SUT 
data. The credibility of the SUT data was enhanced by the use of actual operational hardware, 
wherever possible (e.g., OWS, LGSM, AFATDS), with actual trained operators. VSTARS was 
designed so that the only simulated portion is the simulation of the MTI and SAR radar modes 
within the radar subsystem. Everything else is either integration code or actual E-8C system 
code. The inputs into VSTARS, except for the target data, are normal inputs into the real radar 
processor, and the outputs are the actual radar reports. The radar simulations are parallel 
processes with the radar when live and virtual are mixed and solve the radar equations in order to 
achieve the required fidelity. 


Conclusion for JADS Measure 1-1-0-2. Under normal operations, all input data provided to the 
SUT (VSTARS and LGSM) by the ADS environment were valid. Network performance and 
reliability in delivering data to the SUT are analyzed under JADS Objective 2-1. 


Since the Phase 2 ADS configuration produced valid SUT data, the utility of this configuration 
for Joint STARS DT&E and OT&E was evaluated. 


Utility for DT&E 
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This architecture allows the use of VSTARS to conduct developmental testing of all of the other 
subsystems that comprise Joint STARS, provided that VSTARS is an accurate representation of 
the radar. Obviously, VSTARS can not be used to conduct developmental testing of the radar 
subsystem. | 


As an example, one of the features of the workstation used on the E-8C is an automatic tracker 
(A-tracker). The A-tracker works off radar reports and, when initiated, will automatically track a 
designated formation, providing bearing, speed and number of vehicles. Prior to VSTARS, it was 
necessary to either have a functioning radar (test flight) or a recording of a functioning radar in 
order to test the A-tracker. Test cases were basically limited to those that could be achieved at 
Eglin Air Force Base, Florida, with a minimal number of vehicles traveling under peacetime safety 
restrictions. 


Northrop Grumman is currently developing an annual release of the radar software that will 
incorporate a revised version of the A-tracker software. This provided JADS with the 
Opportunity to ask Northrop Grumman to conduct an ad hoc study, parallel to the normal testing 
of the new A-tracker software, to determine if it would be possible to use VSTARS to test this 
software. 


Numerous problems in the areas of software integration and scenario generation were experienced 
because of the ad hoc nature of the study. Additionally, the V&V of VSTARS had not been 
completed and therefore no results could be used as documentation for the annual release. 
Despite these problems, several lessons were learned from this study relating to utility of 
VSTARS for DT&E. 


— Test cases could be “flown” using VSTARS whenever needed with as many repetitions as 
desired. This was possible without competing for scarce test aircraft and range resources. 


— There was a potential for enormous cost savings. One tester and two computers in a lab vice 
the aircraft, testers, crew and range assets required for a live test. 


— Any conceivable test case could be “flown” in the laboratory without worrying about safety or 
limited assets provided the appropriate scenario generator was available. 


— Bad software could be quickly discarded and new software could be tried the next day. 


— Most importantly, when a live test is flown, as it must be, the testers can be reasonably sure 
that they will get the maximum value from the flight and test conditions. 
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Utility for OT&E 


The utility of this configuration for Jomt STARS OT&E was evaluated by determining which 
measures from the Joint STARS Multiservice Operational Test and Evaluation (MOT&E) Plan’ 
could be supported. Appendix B (available under separate cover from JADS JTF) identifies 
which Joint STARS MOT&E measures could be evaluated using the Phase 2 ADS configuration. 
For comparison, the measures which were addressed during the contingency operations’ were 
also identified. 


Results in Appendix B are summarized as follows. 


— The measures for critical operational issue (COI)-1 (Does Joint STARS perform its tactical 
battlefield surveillance mission?) involving the performance of the E-8C radar in its 
operational environment cannot be evaluated. In Phase 2, a simulator is used to represent the 
E-8C radar subsystem. As a result, the 13 radar measures of performance (MOPs) could not 
be evaluated in the Phase 2 configuration. The Phase 2 configuration can be used to evaluate 
4 out of 5 nonradar MOPs supporting COI-1. However, all three measures of effectiveness 
(MOEs) for COI-1 can at least be partially evaluated using the Phase 2 configuration. 


- As for COI-1, the measures for COI-2 (Does Joint STARS support the execution of attacks 
against detected targets?) involving the performance of the E-8C radar in its operational 
environment cannot be evaluated using the Phase 2 configuration. The Phase 2 configuration 
can be used to evaluate 8 out of 13 nonradar MOPs supporting COI-2. However, all three 
MOEs for COI-2 can at least be partially evaluated using the Phase 2 configuration. 


— The Phase 2 ADS configuration can support evaluation of the single MOE for COI-3 (Does 
Joint STARS provide timely and accurate information to support battlefield management and 
target selection?) and 1 out of 2 MOPs for COI-4. (Can the Joint STARS system be sustained 
in an operational environment?) 


— The Phase 2 ADS configuration can support the evaluation of 6 out of 13 of the additional 
nonradar effectiveness measures. 


— Since the Phase 2 ADS configuration utilizes an actual ground station module (GSM), 8 out 
of 27 suitability MOPs involving the GSM could be evaluated. 


— The Phase 2 ADS configuration does not involve the actual air platform and could address 
only 1 out of 8 human factors measures. : 


' Joint Surveillance Target Attack Radar System (Joint STARS) Multiservice Operational Test and Evaluation 
(MOT&E) Plan, Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, New Mexico, 21 
February 1995. 

* Joint Surveillance Target Attack Radar System (Joint STARS) Contingency Operations Test and Evaluation 
(COT&E) Plan, Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, New Mexico, 1 
December 1995. 
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- The Phase 2 ADS configuration cannot address any of the six software measures since an 
operational Joint STARS software load was not used. 


In summary, the Phase 2 ADS configuration could evaluate 15 out of 45 effectiveness MOPs 
(including two MOPs not evaluated during the contingency operations) and all eight effectiveness 
MOEs. Further, the Phase 2 ADS configuration could be used to evaluate the GSM suitability 
MOPs (8 out of 27 suitability MOPs). However, this configuration would not be useful for 
evaluating the human factors or software measures. 


4.1.2 JADS Objective 1-2. Assess the benefits of using ADS, including DIS, in T&E. 


4.1.2.1 JADS Subobjective 1-2-2. Assess ADS capability to support T&E planning and test 
rehearsal. 


JADS Measure 1-2-2-4. Degree to which pretest exercise of data reduction and analysis 
routines using ADS improved test preparations. 


This measure evaluated the impact of ADS technology on data reduction and analysis routines. In 
support of this test question, the ETE Test analysts conducted interviews with test team members 
involved in data reduction and analysis during the Phase 2 test and test rehearsals. 


The Phase 2 data reduction and analysis methodology was based on procedures successfully used 
in earlier JADS tests. Procedures used during the FI and risk reduction tests, as well as during the 
Phase 2 operational test (OT), were basically unchanged from those used previously. 


Table 2 describes the estimated number of hours spent on rehearsing data reduction and analysis 
procedures in each of the functionality and integration (FI) tests, as well as during the risk 
reduction test. These rehearsals were focused on ensuring that the proper tools and procedures 
were in place for each pretest. Phase 2 pretest activities provided several occasions for JADS 
analysts to practice the data reduction and analysis procedures used later in the operational test. 
The times presented in Table 2 do not include time spent on either data collection or report 
writing. 


Data reduction and analysis activities in Phase 2 were aided by the use of the JADS Analysis 
Toolbox. Developed over the course of earlier JADS tests, the toolbox is a set of software 
routines written in the C++ programming language and integrated into a single user interface. 
The toolbox can be employed to develop tables and plots of PDU data in near real time. After the 
test, toolbox users can replay or get selected data in a text-readable format from JADS logfiles 
and obtain plots and tables of PDU statistics. The PDU statistics, in turn, include the number of 
PDUs received, the number of each type of PDU received, and the rate at which PDUs are being 
received. An important point is that the ETE Test logfiles were larger than earlier JADS test 
logfiles. Modifications were made to the toolbox software to minimize the impact of the size of 
the ETE Test logfiles on file access time, as well as on the amount of memory needed for 
calculations. (For further information on the JADS Analysis Toolbox, please contact the JADS 
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program office or visit the JADS website at http://www.jads.abq.com where it will be available 
until September 2000. After that it will be available at HQ AFOTEC/HO, 8500 Gibson Blvd SE, 
Kirtland Air Force Base, New Mexico 87117-5558 and at the SAIC Technical Library, 2001 
North Beauregard St. Suite 800, Alexandria, Virginia 22311.) 


Table 2. Time Spent on Pretest Data Reduction and Analysis Rehearsals 


FI #1 FI #2 | FI#3 FI #4 | Risk Reduction 
Data Type Test 


‘Latency |S hrs _ 


Availabilit 

Availabilit 
Bandwidth | 
Other OT 
Test 
Measures 


The ETE Test Phase 2 also demonstrated the unique advantages of an ADS environment with 
respect to data reduction and analysis processes. 


— For both ADS and conventional tests, these routines are typically developed prior to the 
actual test events. For non-ADS tests, actual test assets (1.e., flight time) may need to be 
expended in order to produce the data needed to test these routines. In contrast, the JADS 
analysts were able to exercise their data reduction and analysis routines using data from the ΕἸ 
and risk reduction tests accomplished prior to Phase 2 OT testing. This early availability of 
test data enabled the analysts to check out and refine these procedures and prevented the loss 
of critical test data from the actual Phase 2 OT test. 


— If an ADS test team emphasizes the early exercise of data reduction and analysis routines 
during pretest rehearsals, it can reduce or eliminate the loss of critical data and expenditure of 
valuable test assets during subsequent actual testing. 


— In addition, the pretest rehearsals offered testers the opportunity to improve the data 
gathering process. For example, when beginning Phase 2, the ETE Test team intended to log 
all operator actions in the LGSM. As the test progressed, the JADS analysts realized that 
their efforts to obtain data from the LGSM operators were hampering the latter’s abilities to 
provide accurate and timely information to the TAC. As a result, the JADS analysts 
automated most of the data collection and retrieved it at the end of the mission. 
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- The networked nature of an ADS environment provides a built-in means to support data 
management. During the FI and risk reduction tests prior to the Phase 2 operational test, the 
ETE Test team took advantage of this capability and was able to speed the transport of test 
data from the nodes to a central collection and analysis point. 


JADS Measures 1-2-2-5 and 1-2-3-24. Degree to which ADS can be used for tactics 
development prior to test execution. Degree to which ADS can ensure that correct 
techniques and procedures are used. 


These measures determined if the techniques and procedures used by the test participants to 
obtain and disseminate information on target enemy forces could be enhanced in any way using an 
ADS environment. In a broader context, JADS analysts wanted to determine if the ADS test 
environment could be used to correct, refine, and change tactics, techniques, and/or procedures 
prior to exercising a live test. During the rehearsal tests, prior to the Phase 2 operational test, 
JADS analysts recorded the flow of information from the TAC to the LGSM and all fire missions 
initiated by the TAC battle captain and executed using ATACMS. After each of these tests, a 
follow-up interview was conducted with the TAC battle captain to determine the targeting 
process used and to assess the potential of using ADS to improve current techniques and 
processes and develop tactics. 


ADS testing also allowed the operators to use the tactical scenarios and make real-world 
decisions concerning their surveillance and targeting efforts. The JADS analysts noticed a great 
improvement over time in the operators’ abilities to focus on certain aspects of the scenario and in 
their increased use of the tools available to them to obtain more information on the entities they 
were observing. 


The increased test time provided by an ADS environment appears valuable in allowing C4ISR 
system operators and end users to confirm and refine current tactics and to experiment with “what 
if” scenarios and new tactics. This feature will be further examined in the Phase 4 test. 


4.1.2.2 JADS Subobjective 1-2-3. Assess ADS capability to support T&E shortfalls. 


JADS Measures 1-2-3-1, 1-2-3-17, and 1-2-3-28. Degree to which ADS can add test articles 
to test execution, provide for more representative force levels, and increase the number of 
simulation entities. 


Conventional T&E typically suffers from an inadequate number of entities. Typical tests may 
have to be conducted in conjunction with training missions in order to acquire possibly hundreds 
of assets. However, testers will then have only minimal control over test specifics and must base 
their test on the training scenario being executed. 


In contrast, Table 3 shows that the Janus simulation provided thousands of entities for each 
vignette in Phase 2. 
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Phase 2 results imply the following benefits of ADS-based testing. 


-- An ADS environment using validated simulations can provide more realistic force levels than 
those offered by conventional tests, 1.6., levels otherwise unavailable because of cost 
restrictions. 


— C4ISR system testers can tailor the simulation entities operating in the ADS environment to | 
closely reflect the forces expected in operational theaters, thus further increasing the relevance 
of the collected test data. | 


— ADS allows testers to have more control over the specific aspects of the scenarios of interest 
and to expand their test concept and design. A typical constraint to test concept development 
is the number and types of units readily available for a test. For example, a conventional test 
may require the use of a battalion of troops, but because of a prior commitment or cost, these 
personnel may not be available. In contrast, ADS allows testers to create a virtual battalion 
and to test their concepts with a minimum number of personnel and equipment. 


Table 3. Entity Count Per Vignette 


Entities 


JADS Measure 1-2-3-3. Degree to which ADS overcomes performance restrictions. 


ADS is helpful in simulating maneuver scenarios that overcome shortfalls associated with 
traditional testing. This technology increases test robustness by providing the capability for 
adding large numbers of assets. ADS simulations can depict such unsafe conditions as convoy 
vehicles driving across rugged or restricted terrain under wartime conditions. Finally, this 
technology frees the tester from the constraints of environmental restrictions. Without the 
benefits of ADS-enhanced testing, it would be almost impossible to instrument and maneuver a 
corps-size element. 


JADS Measure 1-2-3-4. Degree to which ADS overcomes the traditional T&E shortfall of 
insufficient battlespace. 


ADS technology can easily eliminate the conventional testing disadvantage of insufficient 
battlespace. For example, the National Training Center occupies about one thousand square 


38 


miles. In contrast, the battlespace for Phase 2 of the ETE Test was almost ten times larger. In 
fact, ADS technology is capable of supporting even larger battlespaces. 


JADS Measures 1-2-3-7 and 1-2-3-12. Degree to which ADS overcomes the lack of systems 
for interoperability testing associated with traditional T&E and can increase the number of 
systems for compatibility evaluation. 


There were many incompatibility problems among the fielded real systems that persisted well into 
the risk reduction test. The TAC had never had the opportunity to perform this function, under 
tactical conditions in a doctrinally correct manner, prior to the risk reduction test. Use of the risk 
reduction test ADS environment allowed these incompatibility problems to be identified and 
resolved. 


Phase 2 results show that ADS can link similar systems to perform simultaneous testing and 
training at various locations. The ETE Test Phase 2 used only one LGSM to provide intelligence 
to the TAC at Fort Hood. However, the potential exists for linking numerous LGSMs at different 
military installations and simultaneously conducting the same type of test and training. 


JADS Measure 1-2-3-9. Degree to which ADS increases the number of test events. 


Using ADS, testers can replicate a test, as well as test for longer periods of trme. During the 
Phase 2 test, the ETE Test team was able to test for thirty hours within a 5-day period. A test 
using real assets, and of similar duration, would be very expensive and probably even impossible. 


JADS Measure 1-2-3-31. Degree to which ADS can represent joint/combined operations 
and capabilities. 


ADS can realistically portray joint operations among military elements of the same nation. For 
example, the Phase 2 test employed a mix of Air Force and Army personnel located at different 
facilities successfully simulating a C4ISR system interacting with ground-based units. Given the 
necessary planning and resources, ADS could also represent combined operations between forces 
of two or more allies. 


JADS Measure 1-2-3-36. Degree to which ADS can increase personnel resources. 


The duties Οἱ 811] personnel involved in the setup and execution of the Phase 2 test were examined 
to determine which personnel would not have been required for a traditional test. In addition, the 
number of personnel participating in the test, due solely to ADS-specific requirements, was 
documented. 


The ETE Test Phase 2 results did not provide any set formula for forecasting personnel needs. 
Rather, the exact personnel requirements for C4ISR system testing in an ADS environment 
appears to vary from test to test. With this caveat in mind, and using just the ETE Test team 
requirements, an ADS environment does not appear to add to the personnel needs of a C4ISR 
test. Six JADS personnel are continually involved in ETE Test management and planning, a 
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number equivalent to the staff needed to direct the day-to-day operations of a conventional 
C4ISR test of the same magnitude. Thirteen people were involved in test monitoring and data 
collection at the various ETE Test nodes during the actual execution of the Phase 2 FI, risk 
reduction, and operational tests. Given the distributed nature of C4ISR systems, a similar number 
of people would be needed in a comparable, traditional C4ISR test. Two Network and 
Engineering personnel were needed for network support during the execution of Phase 2, and two 
analysts handled the subsequent data analysis and reduction. However, a conventional C4ISR 
SUT would probably need at least two network engineers because of its distributed nature, and 
maybe more, since it might not be as reliable as the ETE Test Phase 2 network. In addition, the 
C4ISR environment would likewise require a minimum of two data analysts. | 


4.2 JADS Issue 2. What are the critical constraints, concerns, and methodologies 
when using ADS for T&E? 


The ETE Test Phase 2 demonstrates that there are no real technical barriers to using an ADS 
environment to provide a realistic test environment for a C4ISR system. This is due to the high 
reliability of the network architecture underlying an ADS environment and the dramatic increases 
in computer processing and storage capabilities over the past few years. Rather, the key 
constraints to ADS testing are time and money: How soon do you need results? How much are 
you willing to spend on development? 


The following paragraphs discuss constraints and concerns in detail. 


4.2.1 JADS Objective 2-1. Assess the critical constraints and concerns in ADS 
performance for T&E. 


4.2.1.1 JADS Subobjective 2-1-2. Assess network and communications performance constraints 
and limitations. 


JADS Measures 2-1-2-2 and 2-1-3-3. Percentage of ADS trials canceled or otherwise not 
used due to network problems. Percentage of trials in which network connections were lost 
long enough to require trial cancellation. 


For these measures, the network was defined as including all software and hardware used for 
connecting the distributed sites and all loggers and instrumentation used for recording network 
data. NIUs were considered part of the individual simulations and not part of the network. 


For each trial, an execution log was maintained at each node. The data collectors annotated all 
problems encountered, as well as their causes. A test controller log also documented the overall 
status of the network and test trials. In addition, network monitoring tools were used to monitor 
the status of all network links between nodes. Any problems detected by the monitoring tools 
were documented via line printers in terms of a brief explanation of the problem, the time, and the 
link(s) involved. 
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No trials were canceled, or not used, because of network problems. However, two trials were 
postponed when the agency contracted to provide network service inadvertently terminated use of 
the T-1 line at the Northrop Grumman node. The coincidence of a hurricane hitting the Florida 
and Gulf coasts during the same timeframe delayed the reactivation of this link. As a result, the 
_ test was extended two days after the scheduled test period. Table 4 shows the dates of the trials 
and their test times. 


Table 4. ETE Test Phase 2 


| 28Sep | | Postponed to 5 Oct __ 
| 29Sep ΦὋὃ7ὦὺ2ἊἪ _ | Postponed to 6 Oct __ 
| 30Sep(Day1) [| 3 | 7hrs,tmin | 
| 1Oct@ay2) | 4 | 7hrs,4mins | 
| 2Oct(Day3) | 5 hrs, 7 mins | 

-ο-.---] 

LS 


5 Oct (Day 4) 
6 Oct (Day 5) 7 hrs, 15 mins 


JADS Measure 2-1-2-3. Average and peak bandwidth used by link. 


This measure provided an indication of bandwidth use and packet rate during the OT. Although 
bandwidth utilization was not expected to exceed capacity, the utilization rate was documented to 
provide other ADS testers with an indication of the amount of needed bandwidth. The packet 
rate data are also included because of their potential value to other ADS testers. 


Data were collected using the Spectrum’ network analysis tool. Spectrum’ provided the 
capability to study multiple aspects of network link performance, including packet rate and 
percentage of bandwidth utilized. A polling rate of fifteen seconds was used in the collection of 
these data. 


Once all the data were gathered, the JADS analysts consolidated the data by network link. These 
data were then used to calculate daily packet rate and bandwidth values (maximum and average) 
for each link. The bandwidth values were provided by Spectrum’ as the percentage of 
bandwidth available on the T-1 line. A T-1 line has a normal bandwidth of 1.544 megabits per 
second (Mbps). For the ETE Test, some of the bandwidth of the T-1 line was reserved for voice 
traffic, leaving a maximum bandwidth available of 1.344 Mbps. 


Table 5 shows average and maximum performance values for the classified network links 
monitored during the five days of active ETE Phase 2 testing. 


Packet rate and bandwidth utilized remained fairly constant across the five-day test period. The 
packet rate and bandwidth utilization rate experienced over the TCAC-Northrop Grumman link 
averaged approximately 18 packets per second and less than 1%, respectively. The packet rate 
experienced over the Northrop Grumman-Fort Hood link averaged approximately 34 packets per 
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second with an average utilization rate of about 1% of capacity. The maximum packet rate over 
the TCAC-Northrop Grumman link was 120 packets per second resulting in a 20% peak load 
experienced. However, these maximum values appeared to be anomalous, caused by the 
reestablishment of the Janus heartbeat following a crash; the more typical maximum values were 
37 packets per second and 2% bandwidth utilization. The maximum packet rate experienced 
over the Northrop Grumman-Fort Hood link was 102 packets per second resulting in a 5% 
bandwidth utilization rate. These maximum values were consistent from day to day. These data 
for the two links show the relatively small bandwidth utilization experienced during the Phase 2 
test with the maximum packet rate experienced still leaving nominally 95% of the 1.344 Mbps of 
bandwidth available for use. Future ETE tests will use the available bandwidth to facilitate a 
higher heartbeat by sending more frequent entity state PDU updates from the Janus simulation. 


Table 5. Link Performance Characteristics* 


Load Packet Rate 
Average Maximum | Average Maximum 
0.33% 16.93/sec 34.0/sec 

2 0.66% 29.18/sec 98.0/sec 
ΠΤ ae 0.87% 18.60/sec 34.0/sec Ὁ 


OR ee eae τ 


T = TCAC G = Northrop Grumman H = Fort Hood 
* Table refers only to active test time during which PDU loggers were recording data. 


JADS Measures 2-1-2-5, 2-1-2-6, and 2-1-2-7. Percentage of time PDUs were received out 
of order by a network node. Percentage of total PDUs required at a node that were 
delivered to that node. Average and peak data latency between ADS nodes. 


The flow of PDUs to and from each node was recorded using loggers installed as part of the 
network architecture. The loggers specifically recorded the time and order that the PDUs were 
transmitted and received at each node. 
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The raw logger data were transformed and reduced for analysis to determine out of order PDUs, 
duplicate PDUs, lost PDUs, and PDU latency. These data were then used to calculate the 
percentage of out of order, duplicate, and lost PDUs at each node for each vignette and for the 
test as a whole. The minimum, maximum, and mean latency of PDUs were also computed. JADS 
analysts accomplished these calculations using UNIX®-based software tools created by JADS 
programmers. Results for these measures are given in Tables 6 and 7. 


Table 6. Vignette PDU Data 


Sent Percent Rec’d Percent Lost 
: 
99.995 90 0.005% 
ee ee ee 
100% 0% 
| TL siios | outs, 
99.962% 0.038% 
oe ee 3 οὗ 
99. 956% 0.044% 
ee σῦν 
99.970% 0.030% 
ΒΝ toe oe 
100% 0% 


81,695 81,677 18 
99.978 70 0.022% 


fr | se | tosis 
99.909% 0.091% 
sat} Ν 
100% 


94,267 94,254 

94,254 04, 205 
ΠΕ ΣΝ Ὁ τ 
3040 3, 040 
ee | oe ΣΝ 
a pe BE 
99.967 90 0.033% 
a 
π͵ΩΝ ; 


ΠΟ ΝΕ ΝΝ 
99.936% 0.064% 
ee ee ae 
100% 0 
ἮΝ ΤΡ ὴὰ οὐ 
99.915 90 0.025% 
ΒΟ ΒΝ ἫΝ 
99.948 90 0.052% 
a ee 
99. 993% 0.007% 
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W= WSMR T=TCAC Ge=NorthropGrumman δ -Ξ Fort Sill 


Table 7. Vignette Latency Data 


Maximum 
0.145 
0.605 
0.383 
0.139 
1.090 
0.381 


Node A 


3 0.020 0.041 0.079 
0.051 0.058 1.080 
0.030 0.035 0.392 
4 0.016 0.039 0.084 
0.051 0.056 0.580 
0.031 0.036 0.396 
5 0.021 0.038 0.079 
0.050 0.059 0.585 
0.031 0.034 0.361 


0.016 0.039 0.145 
0.050 0.057 1.090 


Total 
W 0.030 0.035 0.396 


W= WSMR T=TCAC G=Northrop Grumman 5 Ξ Fort Sill 


Table 6 shows the PDU data for each vignette by node; there were no duplicate or out of order 
PDUs. Table 7 provides the latency data for the vignettes broken down into the individual 
network links between nodes. These tables indicate the high reliability of the network in passing 
PDUs and the network ability to maintain stable latencies during the Phase 2 test. 


The PDU data in Table 6 show total PDU loss rates of 0.025, 0.052, and 0.007 percent for the 
WSMR-TCAC, TCAC-Northrop Grumman, and Fort Sill-WSMR links, respectively. Note that 
the total loss rate for ESPDUs generated by Janus being delivered from the WSMR node to the 
Northrop Grumman node (the node requiring them) is 0.077 percent (or 295 PDUs lost out of 
382,254 PDUs sent). These loss rates were insignificant and did not affect the validity of the 
Phase 2 OT results. 


The latency data in Table 7 show that the average latency for a given link was very stable over the 
five days of testing, varying by only five percent or less. Note that the latency over the TCAC- 
Northrop Grumman link had maximum values exceeding one second. However, these were only 
transient values and did not significantly affect the average latency (which varied by only about 
3% over the five days). The average total latency for the ESPDU data to travel from the WSMR 
node to the Northrop Grumman node was less than 100 milliseconds and had no effect on the 
validity of the Phase 2 OT results. 
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JADS Measure 2-1-3-6. Average downtime due to ADS network failures. 


This measure identified the impact of network failures on the OT. During Phase 2, logs were kept 
to record all network problems, the start time and duration of the problems, and problem 
resolution. In addition, network monitoring tools were used to monitor the status of all network 
links between the nodes. Any problem detected by the monitoring tools was documented via line 
printers in terms of a brief explanation of the problem, the time, and the link(s) involved. 


During the five days of operational testing, only three network outages were experienced, 
resulting in 8 minutes of downtime (Table 8). 


Table 8. Network Downtime 


Time Time % of Time 

Scheduled for | Network — | Network 

Testing Unavailable | Unavailable 
for ΕΣ 


To τε 


Reason Unavailable 


3 7 hrs, 7 mins 3 min 0.70% Router down at Northrop 
Grumman 


4.2.1.2 JADS Subobjective 2-1-3. Assess the impact of ADS reliability, availability, and 
maintainability on T&E. 


JADS Measures 2-1-3-1 and 2-1-3-5. Percentage of trials delayed, rescheduled, and/or 
redone due to ADS systems (exclusive of network) unavailability. Mean operating time 
between ADS system failures (severe enough to require trial cancellation). 


These measures determined the availability of ADS nodes including the NIUs and the impact of | 
node failures on Phase 2 testing. 


For each trial, an execution log was maintained at each node. The data collectors annotated all 


problems encountered with the ADS systems, along with their causes. A test controller log was 
also maintained to document the overall status of the trials. 
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Once the Phase 2 test network was available, no trials had to be rescheduled. In addition, there 
were no significant delays and only minor ADS system failures. Table 9 lists the reported ADS 
system failures, along with the time needed to resolve these interruptions and their impact on 
testing. 


Table 9. ADS System Failures 


_ Day | Failure | Resolution_| Duration _| Test Time__| Impact on Test 


el a ἈΝΕ ΒθδΝι ἀν τσσται 

unavailable Trial rescheduled 

μα ΘΝ er 
unavailable Trial rescheduled 


One LGSM 7 brs, 
1 console Console 17 mins 


Only one workstation in 


1 min 


crashed at rebooted 


use (minimal impact); no 
PDUs lost 


Fort Hood 


No failures N/A N/A N/A 
LGSM 

Power outage | rebooted 

at once power 

Fort Hood restored Ὁ 

Janus locked 

up at WSMR 


Fire mission coordination 
delayed (minor impact); 

no PDUs lost 

Trial stopped and restarted 
at point of lockup; no PDUs 
lost 

Updates and coordination 
with Fort Hood interrupted; 
no PDUs lost 


21 mins 


Janus 13 mins 


rebooted 


VSTARS 

crashed at 

Northrop 

Grumman 

Janus locked 

up at ΜΕ | Janus 
rebooted 


VSTARS 
rebooted 


10 mins 


Trial stopped, then restarted 
at point of lockup; no PDUs 
lost 
Trial stopped, then restarted 
at point of lockup; no PDUs 
lost 


7 hrs, Updates and coordination 
30 mins 15 mins with Fort Hood interrupted; 
no PDUs lost 
23 mins “οὐ Trial stopped, then restarted 


at point of lockup; no PDUs 

lost 
Extensive risk reduction testing was instrumental in preventing any major ADS system failures 
from affecting the Phase 2 test. Numerous ADS problems were identified and resolved during 
this risk reduction period, minimizing the amount of potential problems for the test. As a result, 
the ADS problems noted in Table 9 had minor impact on Phase 2 testing and caused only brief 
delays or reductions in capability. 


24 mins 


Janus locked 
up at WSMR_ | Janus 
rebooted 


31 mins 


VSTARS 
crashed at VSTARS 
Northrop rebooted 
Grumman 


Janus locked Janus 
up at WSMR_ | rebooted 
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4.2.2, JADS Objective 2-2. Assess the critical constraints and concerns in ADS support 
systems for T&E. 


4.2.2.1 JADS Subobjective 2-2-1. Assess the critical constraints and concerns regarding ADS 
data management and analysis systems. 


JADS Measure 2-2-1-1. Degree to which ADS nodes provide for collection, data entry, and 
quality checking of pre- and post-trial briefing data. 


Quick-look analysis of results was used to support the post-trial briefings. This analysis relied 
primarily on automated data collection at all ETE Test nodes. The data collection tools included 
the JADS logger which collected the PDU log files and a Spectrum’ logger to monitor network 
performance. Data collection tools were attached to the network at each node without any 
impact on network or node performance. At the end of each test day, the data were remotely 
retrieved by the TCAC and the file size checked. This procedure supported timely quick-look 
analysis and test feedback. 


In addition to electronic data logs, manually written logs were kept at each test site and used to 
support post-trial briefings. During the test rehearsals for Phase 2, log sheets were faxed to the 
TCAC at the end of each test day. This procedure proved less than effective, and a daily after- 
action teleconference call was added. This enabled the test controller to discuss and fully 
understand the problems of the day without having to review local log sheets. 


JADS Measure 2-2-1-2. Adequacy of relevant test data storage at ADS nodes. 


The ETE Test analysis requirements drove test data storage needs. The focus of data analysis at 
each site was on network latency, as well as the actual PDU input or data output at each site. The 
need to record PDU traffic at each node required a determination of the data output and reception 
rates at all sites. The largest contributor to ESPDU traffic was the output of the Janus simulation. 
ESPDUs from Janus are a function of the Janus heartbeat and the vignette design. During the 
Phase 2 testing, the Janus heartbeat was set to update all entities every eleven minutes during the 
first hour. In addition, Janus had to output an ESPDU when an entity changed state, i.e., start, 
stop, turn, etc. As a result, the ESPDU output grew as the number of movers increased. The 
ETE Test used five different vignettes, ranging from prehostility with low numbers of movers to 
an active battle vignette with more than 3,000 entities moving at one time. Prior to Phase 2 
testing, the five vignettes were played and the ESPDU output recorded. The maximum file size 
seen during this testing was about fifteen megabytes. To support the data recording as well as file 
storage and local software requirements, the JADS Network and Engineering team installed 4- 
gigabyte hard drives on the SGI Indy at each node. 


During preparations for the Phase 2 test, the Northrop Grumman node required the largest data 
capacity in order to support VSTARS software testing in a stand-alone mode. This testing 
required the playback of PDU files recorded from TRAC-WSMR to VSTARS. All five vignettes 
were played back at various times, and at least five vignette PDU files were stored on the SGI 
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Indy at all times. During actual Phase 2 testing, the ETE Test team found hard drive data storage 
capacity to be more than adequate. 


The development of data storage needs required a full understanding of each node’s requirements. 
Since the cost of hard drive storage has decreased dramatically over the past few years, it was 
cost effective to allow for unexpected growth by significantly exceeding the expected storage 
requirements. 


JADS Measure 2-2-1-4. Ease with which data can be retrieved post trial from a given 
node. 


During Phase 2 of the ETE Test, automated data were collected using PDU loggers at the nodes, 
and operational data were collected using log sheets at each node. The automated data collected 
during the test were retrieved from the JADS test locations by the TCAC using FTP, while JADS 
personnel transported the log sheets to JADS. The automated data were compressed and 
converted for analysis using JADS UNIX®-based tools. 


The automated data retrieval process was very effective. For the ETE Test Phase 2, the 
automated data retrieval process needed about thirty minutes despite the large sizes of the log 
files being converted. Also, there were no problems encountered during any of the retrieval 
efforts. As exemplified by the ETE Test team’s data retrieval process, any ADS data retrieval 
methodology need not be complex. 


4.2.2.2 JADS Subobjective 2-2-2. Assess the critical constraints and concerns regarding 
configuration management of ADS test assets. 


JADS Measure 2-2-2-1. Degree to which test managers can control the configurations of 
ADS participants, the ADS environment data, and ADS networks. 


This measure determined if the test manager could adequately control the test configuration of 
ADS participants, the ADS environment data, and the ADS network both during and between test 
events. The JADS analysts conducted interviews with the test team members involved in 
configuration management on the Phase 2 test. The JADS analysts also monitored the progress 
of test and network configuration from formal integrated product team (IPT) and requirements 
meetings. 


Configuration control for the ETE Test synthetic environment proved to be a challenge. The 
distributed nature of the test made configuration control more complex because of the many 
different organizations involved in the test. Unlike a single-site test effort, the ETE Test involved 
more than half a dozen organizations and two branches of the military. The added difficulty of the 
distributed network made frequent technical meetings and formal IPTs a must. These meetings 
provided the test manager with the ability to track progress and problems with the network 
configuration and data formats. These meetings also provided the forum for the system experts to 
resolve the issues with all those who were affected. 
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This process proved to be effective in controlling the Phase 2 configuration during the test build- 
up and preparation phase. In addition, there were no configuration control problems during the 
OT exectution. 


Configuration control of an ADS test can add significant management issues when compared to 
non-ADS efforts. This added complexity requires a higher level of involvement by the test 
manager, as well as more frequent face-to-face meetings among technical personnel from the 
various ADS test nodes. An ADS test manager’s job would be eased with the aid of an 
integration engineer. 


4.2.3 JADS Objective 2-3. Develop and assess methodologies associated with ADS for 
T&E. 


4.2.3.1 JADS Subobjective 2-3-2. Develop and assess methodologies associated with test 
execution and control for tests using ADS. 


JADS Measure 2-3-2-3. Degree to which protocols, processes, and procedures are needed 
to enable effective centralized test control. 


This measure determined the degree to which protocols, processes, and procedures were needed 
to enable effective centralized test control in a distributed environment. JADS personnel analyzed 
test logs, test team discussions, and after-action reports to determine how well the test control 
procedures worked and how they could be improved. 


The test control procedures used during the Phase 2 test were developed and refined during the 
functionality and integration and risk reduction tests. These procedures included standardized 
checklists addressing the start-up and shutdown of the ETE Test network and the conduct of each 
test trial. The network checklist included procedures to verify network connectivity, data storage 
space, and time server accuracy and to test network transmission prior to test start. The test 
conduct checklist included detailed start-up procedures for each test node. These checklists are 
provided in Appendix A to this report. 


Communication procedures were also developed during the FI and risk reduction tests. Based on 
the experience gained during these tests, the Phase 2 testers were able to use the conference 
phone lines to effectively resolve system failures and to facilitate useful discussions of the after- 
action reports. 


Effective centralized test control was exercised through the test controller located at the TCAC. 
The test controller was responsible for starting and stopping each trial, monitoring the status of all 
nodes and network links, and declaring holds and restarts when necessary. The test controller 
was able to continuously monitor the status of the trials through a combination of network 
monitors in the TCAC and voice communications with key personnel at each site. These 
procedures proved effective for timely detection and resolution of system/network failures. 
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Test control in a distributed computing environment offers many challenges not experienced in 
conventional testing. The designers of a test network should carefully plan for test control, noting 
that good communication is required among all network nodes. Otherwise, poor test control can 
result in inaccurate and untimely information being disseminated throughout the test architecture 
and can waste valuable test time and test assets. 
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5.0 Lessons Learned 


5.1 Technical Lessons Learned 
5.1.1 Simulations 


Simulations when used in ADS testing should be carefully planned and developed. The ADS 
tester must also be in close communication with each organization modifying simulations crucial 
to the test. The ETE Test Phase 2 succeeded because of the relatively high degree of reliability of 
the simulations used in the test. If simulation execution problems had occurred, the ETE Test 
team had arranged for a quick response by the personnel needed to fix those difficulties. 


5.1.2 Interfaces 


Distributed testing often requires linkage among dissimilar facilities, network equipment, and 
simulations. However, careful planning can significantly reduce the potential for difficulties 
arising from network interface problems. As part of their planning, the ETE Test team bought 
standard network equipment for all of the sites. Thus, the configuration of the ETE Test 
environment did not pose any problems during the Phase 2 test. 7 


5.1.3 Networks 


— Time sources must be synchronized. Time must be synchronized from the same time source, 
then validated at each network node to ensure that accurate, synchronized time is recorded at 
each node. The time server used in the ETE Test worked very well, ensuring that all loggers 
were set at the same time and keeping time differences between loggers to a minimum. 
Having the same time at all loggers helped speed up the analysis and allowed for the use of 
automated analysis tools. | 


Data collection methods should be built into the simulation. The ETE Test environment failed 

to take data collection to the next level. Loggers with time stamping were used to record 

PDUs as they were going in and out of simulations. However, data collected from within the 

simulations were not time stamped or synchronized with the logger data. This presented 

considerable problems with the data analysis of the entire C4ISR system. Much of this 
~ analysis was done by hand or accomplished separately from the JADS logger data. 


5.1.4 Instrumentation 


— Special equipment was necessary for ADS network check-out and verification. Special test 
equipment and networking tools will rapidly isolate the specific cause of network and 
ADS/DIS problems. Without the special equipment, troubleshooting would have been 
accomplished by trial and error increasing time, cost, and personnel. In addition, the key 
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Network and Engineering personnel should be trained in the use of the special test equipment 
and networking tools. 


— The flow and transfer of PDUs were critical to the ETE Test. PDU traffic was continuously 
monitored at all nodes to ensure that PDUs were constantly flowing. Since the ETE Test 
manning levels were insufficient to enable watching these monitors at all time, problems that 
interrupted PDU flow were not always immediately noticed. Some type of audio warning 
device might have reduced test time loss from a few minutes to seconds. 


5.2 Infrastructure and Process Lessons Learned 
5.2.1 Procedures 


5.2.1.1 Planning 


— The requirements for an ADS test must be clearly defined early in the test planning phase. 
Detailed planning and coordination are required to ensure a common understanding of all 
requirements, procedures, and test objectives since individual facilities are generally unfamiliar 
with conducting coordinated, distributed T&E tests. 


— Flexibility in planning is essential. When doing something that has never been done before, 
preconceived notions of how the test should be executed will have to change as more 18 
learned. Be open to new ideas, as some of the old ideas from the early stages of an ADS test 
program may become very expensive to bring to fruition. The ETE Test Phase 2 was 
originally slated to have nine scenarios. As the requirements for each scenario increased, their 
development costs also grew. These added costs eventually led to the deletion of the last four 
scenarios. 


-- Plan for the unexpected. Halfway through the ETE Test Phase 2 one of the key T-1 lines was 
inadvertently disconnected. This delayed the test by two days during the most critical portion 
of the test while technicians restored the lines. Travel plans had to be changed and budgets 
were strained. If possible, plan extra days on the end of a test period that can be eliminated if 
all goes well. It is much easier to return early than to stay longer. 


— Minimize the impact of hardware problems. When using a complicated ADS network with a 
vast array of equipment, hardware problems will occur. Plan in such a way that unexpected 
hardware problems do not completely disrupt the test. During the ETE Test Phase 2, steps 
were taken to ensure that hardware problems did not disrupt the test for long periods of time. 
For example, data saves were accomplished frequently. In addition, the network was 
constantly monitored to ensure that hardware problems were fixed as soon as possible. 
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5.2.1.2 Development 


— Use a stepping stone approach to testing where each successive ADS test builds on the 
success of earlier tests. This “test, analyze, fix, test” approach, in concert with structured, 
independent testing of the network, will greatly improve the chances for successful ADS 
testing. 


— Risk reduction testing prior to actual test execution will help test team personnel identify and 
resolve potential ADS system problems. 


— Understand communication requirements. Because of some changes in the test, voice 
communication requirements between the Fort Hood node and the White Sands node were 
dramatically increased. The only way for those nodes to communicate was through the direct 
line to the TCAC. This tied up the direct line for extended periods of time. For the next test, 
another connection will be added for direct communications between Fort Hood and White 
Sands. 


5.2.1.3 Execution 


— Briefings are needed before and after each ADS test. These briefings should include such 
information as the test objectives, telephone numbers to use for test control, the test 
configuration of each facility, instrumentation and data collection requirements, go/no go 
criteria, contingency and backup plans, and test conduct. A briefing checklist should be 
developed and used. 


— Test control procedures should be well rehearsed. When many people are communicating on 
one phone line, a response order should be established and strictly followed to save valuable 
test time. 


- Take advantage of the opportunities provided by ADS technology. At the beginning of Phase 
2, the ETE Test team intended to log all operator actions in the LGSM. As the test 
progressed, the JADS analysts realized their efforts to obtain data from the LGSM operators 
would hamper their activities. Making use of the increase in test time provided by the ADS 
environment, the JADS analysts were able to automate most of their LGSM data collection 
activities and reduce the impact on the LGSM operators. 


5.2.1.4 Evaluation 


— Effective data management is needed. Linked facilities can generate a large volume of data at 
distributed locations. Without careful planning, key data may not be collected and/or 
transmitted to the analysis center, and data collected at the network nodes may not be in a 
useful form for centralized analysis. Before ADS testing, a comprehensive data management 
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plan must clearly identify the data to be collected at each network node, onsite processing of 
the data, and the data to be transmitted to the analysis center. 


Pretest rehearsals are also useful for improving evaluation procedures. The ETE Test team 
improved its data collection and analysis processes as a result of experiences from the 
functionality and integration and risk reduction tests. 


5.2.1.5 Command and Control 


Have test controllers who are extremely familiar with the test and network configuration. The 
Phase 2 test succeeded partly because it had an experienced test controller with the necessary 
tools to evaluate problems and the authority to make meaningful decisions regarding test 
problems. 


Have a centralized test control center. The JADS TCAC is configured to allow for 
convenient, instant communications with all the nodes. It acted as the central point of contact 
between the nodes and for all problems. The test controller kept track of test progress and 
documented any problems that occurred. 


Establish control over resources. Linking various facilities using ADS can require significant 
development of facility interface hardware and software. A well-organized approach by 
experienced personnel enabled the ETE Test team to surmount problems encountered at Fort 
Hood, the most complicated node in terms of getting all necessary hardware established and 
connected before the Phase 2 test. 


Distributed tests require personnel distribution. When many distributed nodes are required for 
the successful completion of a test, personnel will need to be located at these nodes. The 
complexity and input an individual node contributes should guide the assignment of personnel. 
The ETE Test Phase 2 required several people at the Fort Hood, Northrop Grumman and 
TCAC nodes; only one person was needed at the White Sands node. The Fort Sill node used 
only resident personnel. 


5.2.2 Policy 


Network management and troubleshooting must be disciplined and organized with a thorough 
understanding and strong configuration control of the ADS network. This approach enabled 
the ETE Test Phase 2 team to quickly recover from initial pretest network difficulties and to 
go on and achieve five days of successful execution and data collection. 


Flexibility is also needed. When one of the ETE Test network’s T-1 lines was disconnected, 
JADS personnel were able to quickly develop and implement a contingency plan. Upon 
restoration of the T-1 line, the network was soon returned to its original configuration and the 
test continued. 
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5.2.3 Personnel 


Personnel involved in a distributed test should understand the “big picture.” When people are 
geographically separated, 1 becomes easy for them to focus on their own individual portion of 
the test. When problems arise, personne! who understand the entire test and the overall 
network will find solutions much faster. 


6.0 Conclusions/Recommendations 
6.1 Utility 
6.1.1 Utility Conclusions 


6.1.1.1, Enhanced Testing. An ADS environment can enhance the testing of C4ISR systems. 


The Phase 2 test showed that ADS technology can realistically represent a C4ISR system 
enabling the system’s users to collect valid and useful operational information. 


Compared to conventional methods, an ADS environment can realistically test C4ISR 
systems: | 


— in larger battlespaces. It’s possible to connect several Janus simulations together in order 
to multiply the simulated battlespace. 


— with larger numbers of ground-based entities at a much lower cost. 


— with more control over the specific aspects of the scenarios being tested. Using ADS, the 
test director is not at the mercy of a training exercise over which he/she has no direction. 
Rather, the test director can control the simulated entities. 


-- for longer periods of time enabling increased data collection and the ability to analyze and 
improve the data gathering process. 


— under realistic but unsafe conditions, such as convoy vehicles driving across rugged terrain 
under wartime conditions. 


By allowing the simulation of large battlespaces with large numbers of entities, ADS 
technology provides testers with greatly expanded capabilities for test concept and design. 


Testers can use ADS to save time, resources and test personnel man-hours by linking several 
pieces of equipment and/or facilities together for simultaneous testing instead of conducting 
individual tests at different locations. 


ADS technology provides access to elements participating in the test from their normal work 
locations, greatly reducing the additional operational tempo required to support both testing 
and test integration, training and rehearsal. 
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6.1.1.2 DT&E Utility. The Phase 2 ADS architecture can allow the use of VSTARS for 
developmental testing of the JSTARS subsystems, other than the radar subsystems, including 
software. A ground-based ADS environment offers several benefits for DT&E. 


— Multiple repetitions of the same scenario can be performed without competing for scarce test 
aircraft and range resources. 


— The use of ground-based simulations and hardware offers the potential for enormous cost 
savings, as compared to a live test with the aircraft. 


— Any conceivable test case can be “flown” in the laboratory without worrying about safety or 
limited assets provided the appropriate scenario generator is available. 


— Bad software can be quickly discarded and new software could be tried the next day. 


— Most importantly, when a live test is flown, as it must be, the testers can be reasonably sure 
that they will get the maximum value from the flight and test conditions. 


6.1.1.3 Improved Opportunities for Training. fan ADS environment is developed for testing, 
the same environment can easily be modified and transitioned to the training world. ADS 
technology can then improve opportunities for training with a C4ISR system. 


— Conventional training can be limited by the availability of those assets making up the C4ISR 
system’s operational environment. ADS technology, by simulating those key assets, can 
provide longer periods of time for realistic operation of the C4ISR system. 


— C4ISR system operators can take advantage of the additional training time provided by ADS 
technology to confirm current tactics and to test “what if’ scenarios and new tactics. 


— A C4ISR simulation incorporated into an ADS environment can enhance operator training by 
providing useful information on the battlefield surveillance and the targeting process. It can 
also help the battle manager by providing a high-level picture of the entire battlefield and 
aiding in the effective allocation of battle resources. 


— An ADS environment can provide C4ISR system operators with the opportunity to check the 
interoperability and compatibility of their equipment. 


— ADS simulations can help C4ISR system operators familiarize themselves with the maneuver 
tactics of foreign armed forces — valuable experience for possible future deployments. 


— C4ISR system operators can train in an operational environment with multiple assets using the 


capabilities of ADS technology to tie several pieces of equipment and/or facilities together and 
to simulate large numbers of fielded vehicles and associated personnel. In contrast, 
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conventional training would provide far fewer assets in the operational environment, if any at 
all. 


6.1.2 Utility Recommendations 


— Large exercises could use the ETE Test environment to virtually augment the battlefield with 
simulated targets. During Phase 4, this capability will be demonstrated with the integration of 
a live E-8C Joint STARS aircraft into the ETE Test ADS environment. 


— An ADS environment, like the ETE Test environment, is flexible enough to allow for further 
expansion and increased opportunities for testing C4ISR systems. The Janus battlespace can 
be expanded as required. Increasing the number of LGSMs or CGSs would create more 
realistic targeting capabilities. By adding other assets to the environment, such as an 
unmanned aerial vehicle (UAV) or a tactical aircraft simulator, the robustness of the 
environment could be significantly enhanced. 


6.2 Technical 
6.2.1 Technical Conclusions 


— The ETE Test Phase 2 required only a small portion of the available bandwidth. This 
indicates that a much higher packet rate could be applied to ADS testing without causing 
bandwidth problems. The available bandwidth could have been used to facilitate a higher 
heartbeat by sending more frequent updates of entity state PDUs from Janus. During most of 
the Phase 2 test, a 15-minute heartbeat was used during the first hour, followed by a low rate © 
of change-of-state PDUs after the first hour. During the last day of testing, the change-of- 
state PDU rate was dramatically increased. The future ETE Test phases will incorporate the 
higher PDU rate to take advantage of the available bandwidth. 


— The ETE Test network was highly reliable and stable during the Phase 2 test following the fix 
of the T-1 problems experienced on 28 and 29 September. 


— The ETE Test team’s extensive risk reduction testing played an important role in eliminating 
the presence of major ADS system failures during Phase 2 testing. Many ADS problems were 
identified and resolved during this risk reduction period, thus minimizing their potential for 
affecting the actual testing. As a result, the ADS problems noted in Table 9 had only a minor 
impact on Phase 2 testing and caused only brief delays or reductions in capability. 


6.2.2 Technical Recommendations 


— With careful planning and resource management, testers can address the issues associated with 
integrating simulations into an ADS test environment. 
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— Identify the assumptions and limitations associated with those simulations. 


— Budget, schedule, and provide the manpower necessary to develop the simulations. 
Simulation development is typically labor intensive and thus costly. 


— Determine the level of simulation detail needed for the ADS test. Development costs are 
directly related to the level of simulation detail. 


— Identify and provide training for the users of the simulations. 


6.3 Infrastructure 
6.3.1 Infrastructure Conclusions 


— ADS can reduce the number of troops and associated equipment involved in tests because of 
its simulation of fielded forces. However, the ADS infrastructure requires technical personnel 
to set up and execute the tests and to analyze the test results. 


— Highly structured test control is a key ingredient for ADS test success. This test control 
should include formalized procedures with an emphasis on checklists. 


— An ADS test can’t always count on having the personnel requirements for a distant node 
supplied by an organization local to the node. Even if an ADS test is able to employ these 
people, it may then lose them to other activities deemed more important by the local 
organization. 


— An ADS environment necessitates sophisticated instrumentation with rigorous processing 
speed, data storage, and data integration capabilities. This instrumentation can be costly and 
can require trained personnel for its successful operation. 


— Distributed testing typically means distributed personnel and distributed equipment. 
Distributed personnel lead to high travel costs. Equipment located at distant network nodes 
will still require maintenance either through contracts or trips by a network engineering team. 


— ADS analysts must have a well-planned and organized approach to managing the large 
amounts of data produced from ADS testing. 


6.3.2 Infrastructure Recommendations 
— Make every effort to simplify the infrastructure. Time spent in the planning stages of an ADS 


test, with an emphasis on reducing the complexity of the test network, is time well spent. Use 
proven hardware and keep it the same wherever possible. 


61 


- Keep in mind the disadvantages, as well as benefits, of the networked nature of an ADS 
environment. The ADS tester will almost always be dependent upon a telecommunications 
provider. For example, the ETE Test environment was degraded for two days because of the 
inadvertent termination of a T-1 line. 
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APPENDIX A -- Test Procedures 


Al1.0 Test Procedures 


Various types of checklists were used during the execution of the Phase 2 test. The Test Control 
and Analysis Center (TCAC) test controller checklist can be found in Section Al.1, TCAC Test 
Procedures. This checklist was used to ensure network and logger functionality and to provide 
overall test control procedures. Each node (White Sands Missile Range [WSMR], Northrop 
Grumman, and Fort Sill) incorporated the logger functions from the TCAC checklist into their 
own checklist. 


Other checklists were used to direct the operation of various pieces of test equipment. An 
example is included in Section Α1.2, TCAC Plan View Display (PVD) Procedures. 


Section Al.3, WSMR Procedures is representative of the site-specific checklists. WSMR, 
Northrop Grumman and Fort Sill all developed procedures for operation of the End-to-End 
(ETE) Test environment equipment. Only Fort Hood, the only site without a logger, failed to 
develop written procedures. Their procedures were primarily accomplished by the resident 
specialists who have their own procedures. 


Al1.1 TCAC Test Procedures 


The following are the written test procedures used in the TCAC during Phase 2 testing. 


72 HOURS PRIOR TO TEST 


Network Coordinator: 


Date: Test Time: to 
1. Check supplies. 
2. Turn on equipment. 


a. Turn on 3 Barcos (Spectrum [Sun5] on 1, Janus [hp735] on 2, and NetVis [indigo2] on 3). 


Ὁ. Log in as “root” to indigo2 in the TCAC, and indy4 in communications room 1. 
1) From the toolchest, select Toolbox, JADS Toolbox, Monitor, PDU Monitor, PDU 
Statistics, Show Stats to display protocol data units (PDUs). 
2) From the toolchest, select NetVis, NetGraph-ETE to display network traffic. 
3) From the toolchest, select NetTests, Status Check ETE to start and display network 


connectivity tests. (uts in comm rm 1 pings wsmr, ftsill, and fthoodafatads. indigo2, 


pings, grumman, indy3, and sparc5 at Ft Hood). 


c. Inthe TCAC, run Spectrum on the Sun20 (server) and Sun5 (graph) to display Zulu time and 


router status. 
d. Create an empty file “touch /scripts/.go” in grumman, indigo2, and indy4. 
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10. 


11. 


Clear router interfaces. To clear the grumman_router, jads_router, and fthood_router from 
indigo2; and fthood_router, ftsill_router, and wsmr_router from indy4, run: 
“/seripts/clear router ete.” 


Not used. 


Time accuracy. Verify that each site has network time protocol (NTP) running. | 

a. From uts, run “/scripts/check_time’ and verify that the offsets for ftsill and wsmr 
are less than 1 millisecond (ms). 

Ὁ. From indigo2, rlogin to indyl, run “/scripts/check_time.” Verify offsets for 
erumman, indy3, and sparc are less than 1 ms. 


Available disk space. Verify that each logger has at least 600 megabytes (MB) of unused disk 
space available on the /disk2 partition. 
a. From uts, rlogin to ftsill and wsmr, in turn, and 
from indigo2, rlogin to grumman, and indy3, in turn. 
Ὁ. Run “ΔῈ -k” on each machine (including uts) to display the available disk space. Verify that 
each has at least 600 MB available. 


Port settings. Verify that each logger is set to port 3000 and the exercise identification (ID) 1s 0. 
a. From uts, rlogin to ftsill and wsmr, in turn, and 
from indigo2, rlogin to grumman, and indy3, in turn. 

Ὁ. Run “more /scripts/dt_logger’”’ to view the file. Look for the entry: 
“/usr/local/bin/jads_ logger 3000 0 /disk2/logfiles 
/Stestdate” test”Stestnum”’_"Srunnum”_"Ssite”.log” ” entry in 

two places. 


Voice conference net. Verify the net is functional by dialing in from two different phones in the 
TCAC at the same time to establish the net. 


Not used. 


Data collection test a: 

a. From uts, rlogin to ftsill and wsmr, simultaneously, and 
from indigo2, rlogin to grumman, and indy3, simultaneously. 

Ὁ. Start the ftsill, grumman, indy3,and uts loggers using test number “000” and run number ~ 
“a” (Le.- “/scripts/dt_logger 000 a’). 

c. Runthe “/scripts/run_player 3000 /disk2/logfiles/ne_test.1log” 
file on the wsmr machine. : 

4. Determine when run is complete. Stop all loggers (“Ct r1-C’”’). 

e. Check digital communications terminal (DCT) results. Verify reception of 2281 PDUs on 
erumman, indy3, and uts (or indy4) loggers. (No PDUs at ftsill). 


Data collection test b: 

a. From uts, rlogin to ftsill and wsmr, simultaneously, and 
from indigo2, rlogin to grumman, and indy3, simultaneously. 

b. Start the grumman, indy3, uts and wsmr loggers using test number “000” and run number 
“a” (Le. -“/sceripts/dt_logger 000 a’). 
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c. Runthe“/scripts/run_player 3000 /disk2/logfiles/ne_test.log” 
file on the ftsill machine. 

d. Determine when run is complete. Stop all loggers (“Ctr1-C’”). 

e. Check DCT results. Verify reception of 2281 PDUs on grumman, indy3, and uts loggers. 


12: Report the results of the network checks to the test controller. Supervise repairs as necessary to 
prepare equipment for the test sequence. 


PRETEST (DAY OF TEST) 

Network Coordinator: 

Date: Test Time: to 

1. Check supplies. Provide checklists, blank log sheets, file name lists, pens, pencils, scratch paper, 


and 4 millimeter (mm) tape cartridges for the test. 


a Turn on equipment. 
a. Turn on 3 Barcos (Spectrum [Sun5] on 1, Janus [hp735] on 2, and NetVis [indigo2] on 3). 


Ὁ. Log in as “root” to indigo2 in the TCAC, and indy4 in communications room 1. 

1) From the toolchest, select Toolbox, JADS Toolbox, Monitor, PDU Monitor, PDU 
Statistics, Show Stats to display PDUs. 

2) From the toolchest, select NetVis, NetGraph-ETE to display network traffic. 

3) From the toolchest, select NetTests, Status Check ETE to start and display network 
connectivity tests. (uts in Comm Rm 1 pings wsmr, ftsill, and fthoodafatads. indigo2, 
pings, grumman, indy3, and sparcS at Ft Hood). 

c. Inthe TCAC, run Spectrum on the Sun20 (server) and Sun5 (graph) to display Zulu time and 
router status. 


3, Clear router interfaces. To clear the grumman_router, jads_router, and fthood_router from 
indigo2; and fthood_router, ftsill_router, and wsmr_router from indy4, run: 
“/seripts/clear_ router ete.” 


4. Not used. 
i; Time accuracy. Verify that each site has NTP running. 


a. From uts, run “/scripts/check_time’ and verify that the offsets for ftsill and wsmr 
are less than 1 ms. 

Ὁ. From indigo2, rlogin to indyl, run “/scripts/check_time.” Verify offsets for 
grumman, indy3, and sparc are less than 1 ms. 


6. Available disk space. Performed at each logger by the logger operator. 
Vs Port settings. Performed at each logger by the logger operator. 
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10. 


11. 


ΤΖ. 


Join the voice conference net. Both the test controller and the network coordinator (NC) dial 
61143 in the TCAC to establish the conference net. 


Time synchronization. ftsill, grumman, indy3, and wsmr operators check global positioning 
system (GPS) time reception by typing “date” and press Enter on the NC’s mark. Report time to 
NC. 

(NOTE: indy1 is time server for classified, uts is time server for unclassified.) 


Data collection test a: 

a. Cue ftsill, grumman, indy3,and uts operators to start loggers using test number “000” and 
run number “a” (i.e. -““/scripts/dat_logger 000 a”). 

Ὁ. Cue wsmr Operator torun “/scripts/run_player 3000 

/disk2/logfiles/ne_test.log’ file. 

Determine when run is complete. Cue all operators to stop loggers (“Ctr1-c’”). 

4. Check DCT results - Have grumman, indy3, and uts operators verify reception of 2281 
PDUs. (No PDUs at ftsill). 
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Data collection test b: 

a. Cue grumman, indy3, uts, and wsmr operators to start loggers using test number “000” and 
run number “b” (1.e.- “/seripts/dt_logger 000 b’). 

b. Cue ftsill operator torun “/scripts/run_player 3000 

/disk2/logfiles/ne_test.log’ file. 

Determine when run 15 complete. Cue all operators to stop loggers (“Ctr1-C’”). 

d. Check DCT results - Have grumman, indy3, uts, and wsmr operators verify reception of 
2281 PDUs. (No PDUs at ftsill). 
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Report the results of the network checks (items 9-11) to the test controller. 


The pretest phase is now complete. Proceed to the test run phase. 


NOTE: Sometimes the logger process does not terminate on the grumman logger. In that case, run 
/scripts/£ind_logger on the grumman logger to kill the process and delete the old logfile before 
restarting the logger with the same filename. 


TEST RUN 


Network Coordinator: 


Date: 


Lab Time: to 


Start loggers. Obtain the test and run numbers from the test controller and record on the log 
sheet. Operators are cued by the test controller when to start loggers. Record start time on the 


log sheet. 

a. larly in the test run, verify with operators that all loggers are receiving PDUs (number is 
increasing. 

b. Periodically check with operators that all loggers continue to receive PDUs (number is 
increasing). | 
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τ 


Periodically check “bat” phone operation if not used regularly. 

d. Every % hour, run a time accuracy check. From uts, run “/scripts/ check time” to 
check ftsill and wsmr. From indigo2, rlogin to indy1 and run “/scripts/check_time’” to 
check indy1, grumman, and tcacindy. Time offsets should be <1 ms. 

6. Keep written event log. 


z: Stop loggers. Loggers stop recording data when directed by the test controller (“Ctrl-C”). 

a. Record the stop time and the total number of PDUs from each logger on log sheet. 

b. Confirm that the required data have been logged. From uts, rlogin to ftsill and wsmr and 
run “ls -1 /disk2/logfiles.” From indigo2, rlogin to indy3, and grumman and 
run“ls -1 /disk2/logfiles.” Check the file sizes; the filename is 
“nunddyy_test#-run#_loggername.log.” 


3; Subsequent runs. When additional runs are required, repeat steps 1 and 2 for each run. 


The test run phase is now complete. Proceed to the post-test phase. 


POST TEST (DAY OF TEST). 


Network Coordinator: 


Date: Test Time: to 
1. Remote file capture. Consolidate, compress, and copy the test run logger files from each remote 
site. | 
a. For classified data, rlogin to each logger (grumman and indy3), in turn, from teacindy in the 
TCAC, or 


For unclassified data, rlogin to each logger (ftsill, wsmr, and uts), in turn, from uts. 

Ὁ. If only 1 file for the day exists in the logger at a site, skip to step c. If more than 1 log file for 
the day exists at a site, consolidate them by using the command 
“tar cv£ mmddyy_sitename.log.tar mmddyy*.log” 
where * is the wildcard character that includes all the files for that day for that site name. 
(e.g.,- “tar cv£ 040798 wsmr.log.tar 040798*.log” ). 

c. Compress the single log file (e.g.,- “compress 040798 wsmr.1log”) or the tar file 
from step Ὁ (“compress 040798 wsmr.tar’). 

4. On tcacindy, run “/scripts/rcep_etefile” to copy the tar'd and compressed classified 
files (“mmddyy_sitename.log.Z’’) from both grumman and indy3 loggers to 
tceacindy:/disk2/ete/mmddyy/. 

e. Onuts, run “/scripts/rcp_etefile” to copy the tar'd and compressed unclassified 
files (“mmddyy_sitename.log.z”) from ftsill, uts, and wsmr loggers to 
uts:/disk2/ete/mmddyy/. 

f. Copy the unclassified files from step e to 4mm tape and activate the write protect feature. 
Move the tape to teacindy and copy the unclassified files from the tape to /disk2/ete/mmddyy/ 
(1.e., from the ete directory, run tar xv to extract the files from the tape to the hard drive). 
Make sure the write protect feature is ON. 


2. Backup tapes. 
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a. Create a backup tape of the files in teacindy:/disk2/ete/mmddyy/ using either the “tar cv 
mmddyy” command while in the ete directory (or the tape tool on the teacindy desktop). 


Ὁ. Verify the backup using either the “tax tv’ command or the tape tool. 

c. Remove the tape from the drive and label it. 

4. Repeat a, Ὁ, and c to create a duplicate tape. 

e. Deliver both tapes to the ETE Test team representative. 
Ἂς Delete the data collection test and the backed-up log files from /disk2/logfiles/ on all loggers. 
4. On the last day of testing, delete the file “/scripts/.go” in grumman, indigo2, and indy4. 
OF Logoff from logger. Turn Off the monitor, but leave the central processing unit (CPU) On!!! 
6. Participate in mission debrief, if applicable. 


A1.2 TCAC Plan View Display (PVD) Procedures 


The following procedures were used to initiate test monitoring with the Janus plan view display 
program. This is representative of the specific checklists developed to aid in the operation of test 
equipment. 


Functionality/Integration Test Checklist 


(TCAC-PVD) 
Date: 
Scenario: 
Test Start Time (z): Test Stop Time (2): 
Scenario Start Time: Scenario Stop Time: 


1 ETE | Power on the hp735 monitor. 
Log on to the hp735 as hovey. 


2 ETE | From the xterm window that Use this alias to start | 
appears, type pvd, and hit enter. | Janus plan view display. | 


3 ETE | From the Janus plan view 
display menu, verify the 
parameters for the run: 
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Workstation J 

Terrain File Use the correct terrain file 
Screen File and screen file for the 
Symbol File 3 vignette. 

Symbol Size 10 

Terrain File Meridian 45 

Exercise ID BLANK 

Map Spheroid J 

Mode BLANK 

Terminate this Run N 


and hit keypad enter. 

Wait until the PVD terrain and | Last message: Opening 
combat systems databases are file 

loaded. | _/jads_ete/trn/TSCRN . 
DAT 


Double click the 
Analyst_Workstation_WS1I icon 
to bring up the scenario 
window. 

| From the 
Analyst_Workstation_WS1 


scenario window, functions 
menu, | 
left click Draw CAC. 
From the Places command and 
Analyst_Workstation_WS1 control overlays on the 
scenario window, CAC File scenario box. 
menu, select the CAC file 
number to display. 
Left click increases number, and 
right click decreases number. 
Left click Add to display the 
CAC. 
From the Ready to receive and 
Analyst_Workstation_WS 1 display DIS PDUs. 
scenario window, function 
menu, 
left click Display. 
From the 
Analyst_Workstation_WS1 
scenario window menu: 
Used as necessary to 
Left click any tick on the zoom | zoom in/out of the 
in/out menu, then select the scenario box. 


desired zoom point on the 

scenario box. 
Used as necessary to add 

Left click CAC. or remove the command 
and control overlays 
which have been added in 
step 7. 


Left click Display. Used as necessary to start 
or stop Janus plan view 
display from receiving 
PDUs. 

Left click Clear. 

Used as necessary to clear 
any text or information 
displayed on the scenario 
box. : 


NOTE: A particular 


function is active when 
hi hted. 


From the Shuts down PVD. 
Analyst_Workstation_WS1 

scenario window, 

right click End. 

Minimize the In the xterm that remains, 
Analyst_Workstation_WS1 verify this message: 
scenario window. 


From the Closes the scenario 
Analyst_Workstation_WSI window. 

icon, 

right click and choose close. 


| Shut Down Test _. 


] ETE | Left click EXIT from the HP Signs off the hp735. 
VUE front panel. 
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᾿ ETE | Left click Continue logout from | Confirms desire to log rSigagaee 
the dialog box. out. 
Power off the hp735 monitor. Oe oe 


A1.3 WSMR Procedures 


This checklist is representative of the individual site checklists. It incorporates the logger 
functionality and the site specific actions required by the operator(s). These are maintained by the 
site specialists and updated as changes are required. 


Functionality/Integration Test Checklist 


(WSMR) 
Date: 
Scenario: 
Janus File: 
Indy File: 
Test Start Time (z): Test Stop Time (z): 
Scenario Start Time: Scenario Stop Time: 


i ETE | Verify operation of hotlink Enables secure and 
and | phone. If no go, contact N&E unclassified voice : 
N&E | to fix the network. communications. 2 
2 TE | Verify that WSMR indy and the Ἶ 


Ε 
and | WSMR hp715 are on the JADS | network is operational. 
WSM_ | ETE network. 


R 
3 N&E | Verity N&E has cleared and Clears router interface 
reset routers. cards. ' 


4 ETE | Power on the WSMR monitor. | Restarts the indy. 
Log on to WSMR as dislog. 
From a Unix shell window as 
su, run /scripts/restart. 

5 


ETE | After a successful restart, log Signon is used for 

on to wsmr as dislog. checking network 
communications and 

logging PDU data. 
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From a Unix shell window as 
SU, 


run /scripts/ping_test to get 
ping statistics for each remote 
Site. 


Verifies that each network 
link is operational. 3% 
loss at Fort Sill and uts is 
normal. 


From ἃ Unix shell window as Displays the offset from 
su, run /scripts/check_time. uts. Should be less than 1 | 
ms. 

From a Unix shell window as su | Verifies sending 2281 
and at the test controller’s PDUs and receiving the 
direction, same number of PDUs at 
run /scripts/run_player 3000 each remote site. 
/disk2/Nogfiles/ne_test.log to 
check ability to send PDUs to 
each remote site. 
From a Unix shell window as su | Verifies receiving 2281 
and at the test controller’s PDUs from a remote site. 
direction, 
run /scripts/dt_logger 

to check ability to 
receive PDUs from a remote 
site. At the test controller’s 
direction, hit Ctrl-C to end the 
logfile. 
From a Unix shell window, Verifies that PDU_rate = 
cd /usr/local/bin and 0. Ensures that there 
run ./display_pdu_rate. aren’t any DIS 
Select port 3000 0. communications before 
Left click start. the start of testing. 
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From a Unix shell window as su | Script that runs the JADS 
on WSMR, run 
/scripts/dt_logger 


Verify the logfile name as Opens port 3000 to listen 

/disk2/Nogfiles/ and log all DIS 

mr.log and port 3000. communications. Writes 
to the listed logfile. 


Start Janus 


ETE | Power on the c180 monitor. ois 
Log on to the c180 as JADS. ει 
ETE | From δὴ hpterm window, type | Use this executable to 
jJanus.exe, and hit enter. start Janus 


Left click PE (Program Brings up the Program 
Execution) from the Janus User | Execution menu. 
Options menu. 


ETE | Left click JE (Janus Execution) | First step in defining the 
from the Program Execution scenario. 
menu. 


Type desired scenario number | Tells Janus which scenario |. 
for the run, and hit to run. 

enter. 

Type run number 1, and hit 

enter. 


Hit enter again to continue. Ready to continue. , 
ETE | Verify that 7 is entered. Hit Use a normal run. mee 
enter one more time. 


ETE | From the Janus Runtime Verifies time of day. 
Screens menu, 
left click IT. 
Verify time of day is correct for 
the vignette, and hit keypad 
enter. 


a 
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ETE 


From the Janus Runtime 
Screens menu, 
left click 22. 


Verify that there is a setup for: 


WS Number I,and Side J, 
and hit keypad enter. 
From the Janus Runtime 
Screens menu, 

left click 66. 

Verify the DIS operational 
parameters for the run: 
Janus side 1 

DIS side 2 

DIS COMM calls/sec 


Units processed/COMM call 


Terrain File Meridian (+E) 45 
Heartbeat(s) 


Dead Reckoning Threshold 


999 

Site TRAC-WSMR 23 
Host CPU HP 4 
Exercise JADS-ETE 4 
DIS version transmit 4 
DIS version receive 4 


and hit keypad enter. 
Left click JJ (Begin Janus) 


from the Janus Runtime Screens 


menu. 

Wait until the Janus scenario 
loads. Verify: 

Scenario number 

Total number of units 


Double click the side/ icon to 
bring up the scenario window. 
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Verifies that a controller 
workstation has been 
configured. 


Verifies DIS parameters. 
Calculate the new 
heartbeat as follows: 


CxRxH<T 


where 

C = calls/sec, 

R = units/call, 

H = heartbeat, and 

T = total number of units 
in scenario 


Loads the Janus scenario. 


This brings up the 
scenario window which 


allows a Janus operator to | 


interact (game) the 
exercise. 


Run Scena 


] ETE | From the side! scenario DIS button highlights. 
δ Opens DIS 
left click DIS. : communications. 
2 ETE | From the side! scenario First step in running a 
window, Janus scenario. 
left click START. 


3 ETE | Minimize the Janus scenario Ready to continue the 
window (sidel). Type rrinthe | Janus run. 
Janus window, and hit enter. 


Type n and hit enter. No planned save. on 
5 ETE | Hit enter again. Default checkpoint | | 
frequency. β 


ETE | Double click the Janus scenario | Verifies scenario 
window (side/). movements and a running 


time of day counter. 


ἶ Stop Scenario 
DIS button unhighlights. 
Closes DIS 


1 ETE | From the side/ scenario 
left click DIS. communications. 
2 ETE | From the 5166] scenario Brings up options menu. 

window, 

7 right click ADMIN. ἢ 
Right click 2 times. Completely closes Janus. “τῷ, β | 
5 ETE | Left click EXIT from the HP Sign off the hp715. | 
VUE front panel. PE gs 


7] 


Shu Down Tes Heise 


Power ΕΣ ΩΣ the c180 monitor, ib me 
ΕΣ ΩΣ shutdown CPU. ἑ . 


2 ee Make sure that JADS N&E Ensures data integrity. 

FTP 

hoe /disk2/ogfiles/ ws 
mr.log 
back to JADS and place the file 
in 
/usr/testdata2/ogs/ete/DDMM 
YY 


This file will be analyzed 
ETE | Power off the wsmr monitor. 


by JADS analysts. 
β : ͵ 
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Appendix B 
Pages 79-86 
intentionally removed | 


APPENDIX C -- Glossary 


A 


Accreditation. See: distributed simulation accreditation, model/simulation accreditation. 

Accuracy. The degree of exactness of a model or simulation relative to an established standard 
with high accuracy implying low error. [DIS] 

Activity. An event that consumes time and resources and whose performance is necessary for a 
system to move from one event to the next. [DIS] 

Advanced Distributed Simulation (ADS). A set of disparate models or simulations operating in 
a common synthetic environment. The ADS may be composed of three modes of simulation: 
live, virtual and constructive, where the latter can be seamlessly integrated within a single 
exercise. See also: live simulation; virtual simulation; constructive simulation. [DIS] 

Aggregate. An activity that combines individual entities into a singular entity. Contrast with: 
disaggregate. [DIS] 


B 


Battlespace. The three-dimensional battlefield. [DIS] 

Benchmark. (v) The activity of comparing the results of a model or simulation with an accepted 
representation of the process being modeled. (n) The accepted representation of the modeled 
process. [DIS] 

Bit. The smallest unit of information in the binary system of notation. [TEEE 1278.1] 

Broadcast. A transmission mode in which a single message is sent to all network destinations, 


1.e., one-to-all. Broadcast is a special case of multicast. Contrast with: multicast; unicast. 
(IEEE 1278.2] 


C 


Compatible. Two or more simulations are DIS compatible if (1) they are DIS compliant, and (2) 
their models and data that send and interpret PDUs support the realization of a common 
operational environment among the systems (coherent in time and space). Contrast with: 
compliant, interoperable. [DIS] 

Compliant. A simulation is DIS compliant if it can send or receive PDUs in accordance with 
IEEE Standard 1278 and 1278 (working drafts). A specific statement must be made 
regarding the qualifications of each PDU. Contrast with: compatible, interoperable. [DIS] 

Conceptual Model. A description of the content and internal representations which are the user's 
and developer's combined concepts of the exercise. It includes logic and algorithms and 
explicitly recognizes assumptions and limitations. [DIS] 

Constructive Simulation. Models and simulations that involve simulated people operating 
simulated systems. See Also: war games; higher order model (HOM). [DIS] 

Continuous Model. (1) A mathematical or computational model whose output variables change 
in a continuous manner; that is, in changing from one value to another, a variable can take on 
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all intermediate values. For example, a model depicting the rate of air flow over an airplane 
wing. Syn: continuous-variable model. (2) A model of a system that behaves in a continuous 
manner. Contrast with: discrete model. [DIS] 

Continuous Simulation. A simulation that uses a continuous model. [DIS] 

Continuous-Variable Model. See: continuous model. [DIS] 

Control Station. (1) A facility which provides the individual responsible for controlling the 
simulation and the capability to implement simulation control as protocol data units (PDUs) 
on the distributed interactive simulation (DIS) network. 

Syn: simulation - management station. [DIS] 


D 


Data. Representation of facts, concepts, or instructions in a formalized manner suitable for 
communication, interpretation or processing by humans or automatic means. [DIS] 

Database. A collection of data organized according to a schema to serve one or more 
applications. [DIS] | 

Data Certification. The determination that data have been verified and validated. (1) Data 
producer certification is the determination by the data producer that data have been verified 
and validated against documented standards of criteria. (2) Data user certification is the 
determination by the application sponsor or designated agent that data have been verified and 
validated as appropriate for the specific M&S usage. [DIS] 

Data Logger. A device that accepts protocol data units (PDUs) from the network and stores 
them for later replay in the same time sequence as the PDUs were originally received. See 
also: protocol data unit (PDU). [IEEE 1278.3] 

Data Validation. The documented assessment of data by subject area experts and comparison to 
known or best-estimate values. (1) Data producer validation is that documented assessment 
within stated criteria and assumptions. (2) Data user validation is that documented 
assessment of data as appropriate for use in an intended M&S. [DIS] 

Data Verification. The use of techniques and procedures to ensure that data meet specified 
constraints defined by data standards and business rules. (1) Data producer verification is the 
use of techniques and procedures to ensure that data meet constraints defined by data 
standards and business rules derived from process and data modeling. (2) Data user 
verification is the use of techniques and procedures to ensure that data meet user specified 
constraints defined by data standards and business rules derived from process and data 
modeling and that data are transformed and formatted properly. [DIS] 

Data Verification, Validation, and Certification. The process of verifying the internal 
consistency and correctness of data, validating that they represent real world entities 
appropriate for their intended purpose or an expected range of purposes, and certifying them 
as having a specified level of quality or as being appropriate for a specified use, type of use, or 
range of uses. The process has two perspectives: producer and user process. See: data 
validation, data verification, and data certification. [DIS] 

Dead Reckoning. See: remote entity approximation. 

Deagegregate. See: disaggregate. [DIS] 
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Distributed Interactive Simulation (DIS). A synthetic environment within which humans may 
interact through simulation(s) at multiple sites networked using compliant architecture, 
protocols, standards, and databases (DoDD 5000.59P) 


E 


Electronic Battlefield. See: synthetic environment. [DIS] 

Entity. Any component in a system that requires explicit representation in a model. Entities 
possess attributes denoting specific properties. See: simulation entity. [DIS] 

Environment. (1) The texture or detail of the domain, such as cities, farmland, sea states, etc. 
(2) The external objects, conditions, and processes that influence the behavior of a system 
(such as terrain relief, weather, day, night, terrain cultural features, etc.) [DIS] 

Event. (1) An occurrence that causes a change of state in a simulation. See also: conditional 
event; time-dependent event. (2) The instant in time at which a change in some variable 
occurs. [DIS] 

Event-Driven Simulation. See: event-oriented simulation. [DIS] 

Event-Oriented Simulation. A simulation in which attention is focused on the occurrence of 
events and the times at which those events occur; for example, a simulation of a digital circuit 
that focuses on the time of state transition. Syn: event-driven simulation; event-sequenced 
simulation. [DIS] 

Event-Sequenced Simulation. See: event-oriented simulation. [DIS] 

Exercise. (1) One or more sessions with a common objective and accreditation. (2) The total 
process of designing, assembling, testing, conducting, evaluating, and reporting on an activity. 
See: simulation exercise. Syn: experiment, demonstration. [DIS, IEEE 1278.3] 


F 


Fidelity. (1) The similarity, both physical and functional, between the simulation and that which 
it simulates. (2) A measure of the realism of a simulation. (3) The degree to which the 
representation within a simulation is similar to a real-world object, feature, or condition in a 
measurable or perceivable manner. See also: model/simulation validation. [DIS, [EEE 1278.1] 

Field. (1) A series of contiguous bits, treated as an instance of a particular data type, that may be 
part of a higher level data structure. (2) An external operating area for actual vehicles or live 
entities. See: field instrumentation. [DIS, IEEE 1278.1] 


G 


Graphical Model. A symbolic model whose properties are expressed in diagrams. For example, 
a decision tree used to express a complex procedure. Contrast with: mathematical model; 
narrative model; software model; tabular model. [DIS] 

Ground Truth. The actual facts of a situation without errors introduced by sensors or human 
perception and judgment. [DIS] 
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Human-in-the-Loop Model. See: interactive model. 

Human-Machine Simulation. A simulation carried out by both human participants and 
computers, typically with the human participants asked to make decisions and a computer 
performing processing based on those decisions. [DIS] 


Interactive Model. A model that requires human participation. Syn: human-in-the-loop model. 
[DIS] 

Interoperable. Two or more simulations are DIS interoperable for a given exercise if they are 
DIS compliant, DIS compatible, and their performance characteristics support a fair fight to 
the fidelity required for the exercise. Contrast with: compatible, compliant. [DIS] 

Interoperability. (1) The ability of a set of simulation entities to interact with an acceptable 
degree of fidelity. The acceptability of a model is determined by the user for the specific 


purpose of the exercise, test, or analysis. (2) The ability of a set of distributed interactive 
simulation applications to interact through the exchange of protocol data units. [DIS] 


L 


Live Entity. A perceptible object that can appear in the virtual battlespace but is unaware and 
non-responsive (either by intent, lack of capability or circumstance) to the actions of virtual 
entities. See also: field instrumentation. Contrast with: live instrumented entity. [DIS] 

Live Instrumented Entity. A physical entity that is in the real world and can be represented in 
the distributed interactive simulation (DIS) virtual battlespace which can be manned or 
unmanned. The live instrumented entity has internal and/or external field instrumentation (FD) 
devices/systems to record and relay the entity’s surroundings, behavior, and/or reaction to 
events. If the FI provides a two-way link, the events that affect the live instrumented entity 
can be occurring in the virtual battlespace as well as the real world. See also: field 
instrumentation, live entity. [DIS] | 

Local Area Network (LAN). A class of data network which provides high data rate 
interconnection between network nodes in close physical proximity. [[EEF 1278.3] 


M 


Measure of Performance (MOP). Measure of how the system/individual performs its functions 
in a given environment (e.g., number of targets detected, reaction time, number of targets 
nominated, susceptibility of deception, task completion time). It is closely related to inherent 
parameters (physical and structural) but measures attributes of system behavior. See also: 
measures of effectiveness (MOB). [IEE 1278.3] 
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Model. ([) An approximation, representation, or idealization of selected aspects of the structure, 
behavior, operation, or other characteristics of a real-world process, concept, or system. 
Note: Models may have other models as components. (2) To serve as a model as in (1). (3) 
To develop or use a model as in (1). (4) A mathematical or otherwise logical representation of 
a system or a system’s behavior over time. [DIS] 

Model/Simulation Accreditation. The official certification that a model or simulation is 
acceptable for use for a specific purpose. See also: distributed simulation accreditation. 
Contrast with: model/simulation validation, model/simulation verification. [DoDD 5000.59] 

Model/Simulation Validation. The process of determining the degree to which a model is an 
accurate representation of the real world from the perspective of the intended use(s) of the 
model. See also: distributed simulation validation, fidelity. Contrast with: model simulation 
accreditation, model simulation verification. [DoDD 5000.59] 

Model/Simulation Verification. The process of determining that a model implementation 
accurately represents the developer's conceptual description and specifications. See also: 
distributed simulation verification. Contrast with: model simulation accreditation, model 
simulation validation. [DoDD 5000.59] 


N 


Network Filter. A system to selectively accept or reject data received from the network. [DIS] 

Network Node. A specific network address. See: node. Contrast with: processing node. [DIS] 

Node. A general term denoting either a switching element in a network or a host computer 
attached to a network. See: processing node; network node. [IEEE 1278.1, IEEE 1278.2] 


O 


Operational Environment. A composite of the conditions, circumstances, and influences which 
affect the employment of military (or other) forces and the decisions of the unit commander or 
person in charge. [DIS] 


P 


Platform. A generic term used to describe a level of representation equating to vehicles, aircraft, 
missiles, ships, fixed sites, etc., in the hierarchy of representation possibilities. Other 
representation levels include units (made up of platforms) and components or modules (which 
make up platforms.) [DIS] 

Protocol Data Unit (PDU). A DIS data message that is passed on a network between simulation 
applications according to a defined protocol. [IEEE 1278.1] 


9] 


R 


Real Time. In modeling and simulation, simulated time advances at the same rate as actual time; 
for example, running the simulation for one second results in the model advancing time by one 
second. Contrast with: fast time, slow time. [DIS] 

Resolution. (1) The degree to which near equal results values can be discriminated. (2) The 
measure of the ability to delineate picture detail. [DIS] 


S 


Scenario. (1) Description of an exercise (initial conditions). It is part of the session database 
which configures the units and platforms and places them in specific locations with specific 
missions. (2) An initial set of conditions and time line of significant events imposed on trainees 
or systems to achieve exercise objectives. See: field exercise. [DIS, IEEE 1278.3] 

SIMNET (Simulator Networking). The prototype distributed simulation upon which DIS was 
based. [DIS] 

Simulate. To represent a system by a model that behaves or operates like the system. See also: 
emulate. [DIS] 

Simulated Time. Time as represented within a simulation. Syn: virtual time. See also: fast time; 
real time; slow time. [DIS] 

Simulation. (1) A model that behaves or operates like a given system when provided a set of 
controlled inputs. Syn: simulation model. See also: emulation. (2) The process of developing 
or using a model as in (1). (3) An implementation of a special kind of model that represents at 
least some key internal elements of a system and describes how those elements interact over 
time. [DIS] 

Simulation Environment. (1) Consists of the natural physical environment surrounding the 
simulation entities including land, oceans, atmosphere, near-space, and cultural information. 
(2) All the conditions, circumstances, and influences surrounding and affecting simulation 
entities including those stated in (1). [DIS] 

Simulation Exercise. An exercise that consists of one or more interacting simulation 
applications. Simulations participating in the same simulation exercise share a common 
identifying number called the exercise identifier. These simulations also utilize correlated 
representations of the synthetic environment in which they operate. See: live simulation. 
(IEEE 1278.1, IEEE 1278.2] 

Simulation Fidelity. Refers to the degree of similarity between the simulated situation and the 
operational situation. [IEEE 1278.3] 

Simulation Time. (1) A simulation’s internal representation of time. Simulation time may 
accumulate faster, slower, or at the same pace as real time. (2) The reference time (e.g., 
universal coordinated time) within a simulation exercise. This time is established ahead of 
time by the simulation management function and is common to all participants in a particular 
exercise. [DIS, IEEE 1278.1] 
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Simulator. (1) A device, computer program, or system that performs simulation. (2) For 
training, a device which duplicates the essential features of a task situation and provides for 
direct practice. (3) For distributed interactive simulation (DIS), a physical model or simulation 
of a weapons system, set of weapon systems, or piece of equipment which represents some 
major aspects of the equipment’s operation. [DIS] 

Site. (1) An actual physical location at a specific geographic area, e.g., the Fort Knox Close 
Combat Test Bed (CCTB). (2) A node on the network used for distributed simulation such as 
the Defense Simulation Internet (DSI) long haul network. (3) A level of configuration 
authority within a DIS exercise. [DIS] 


ν᾿ 


Validation. See: data validation, distributed simulation validation, face validation, 
model/simulation validation. [DIS] 

Verification. See: data verification, distributed simulation verification, model/simulation 
verification 

Verification and Validation (V&V) Proponent. The agency responsible for ensuring V&V is 
performed on a specific model or simulation. [DIS] 

Vignette. A self-contained portion of a scenario. [DIS] 

Virtual Battlespace. The illusion resulting from simulating the actual battlespace. [DIS] 


W 


War Game. A simulation game in which participants seek to achieve a specified military 
objective given pre-established resources and constraints; for example, a simulation in which 
participants make battlefield decisions and a computer determines the results of those 
decisions. See also: management game. Syn: constructive simulation; higher order model 
(HOM). [DIS] 

Wide Area Network (WAN). A communications network of devices which are separated by 
substantial geographical distance. Syn: long haul network. [IEEE 1278.3] 
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ANIU 
ARIES 
ARPA 
ASAS 
ATACMS 
A-tracker 
ATWS 
Bde 

Bn 

C4] 
C4ISR 


CAMPS 


CBS 
CDP 
CEP 
CGS 

Co 

COI 
COT&E 
CPU 
D&SA BL 
DCT 
DDSA 
DIS 
DMAP 
DoD 
DT&E 
ECCM 
ESPDU 
ETE 
EW 
FDC 


APPENDIX D -- List of Acronyms 


Ath Infantry Division, Fort Hood, Texas 

air defense artillery 

advanced distributed simulation 

Advanced Field Artillery Tactical Data System 

Air Force base 

a mature self-protection jammer system; an electronic countermeasures 
system with reprogrammable processor developed by Georgia Technical 
Research Institute | 

amplitude modulation 

air network interface unit 

Advanced Radar Imaging Emulation System 

Advanced Research Projects Agency 

All Source Analysis System 

Army Tactical Missile System 

automatic tracker 

Advanced Technology Work Station 

brigade 

battalion 

command, control, communications, computers and intelligence 
command, control, communications, computers, intelligence, surveillance 
and reconnaissance 

Compartmented All Source Analysis System (ASAS) Message Processing 
System 

corps battlefield simulation 

central data processor 

circular error probability 

common ground station 

company 

critical operational issue 

contingency operations test and evaluation 

central processing unit 

Depth and Simultaneous Attack Battle Lab 

digital communications terminal 

deputy director, system assessment 

distributed interactive simulation 

data management and analysis plan 

Department of Defense 

developmental test and evaluation 

electronic counter-countermeasure 

entity state protocol data unit 

End-to-End Test 

electronic warfare 

fire direction center 
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ΕῚ 
ΕΜ 
FTI 
FTP 
GHQ 
GPS 
GSM 
GSMR 
HF 
HLA 
hrs 
{CD 
{D 
IEEE 
IPT 
JADS 
Janus 
Joint STARS 
JPO 
JT&E 
JTF 


MTT 
N&E 
NC 


NET Visualizer™ 


NIU 

NTP 
O&C 
OSD 


functionality and integration 

frequency modulation 

fixed target indicator 

file transfer protocol 

general headquarters 

global positioning system 

ground station module 

Ground Station Module Replicator, Fort Huachuca, Arizona 
high frequency 

high level architecture 

hours 

interface control document 

infantry division; identification 

Institute of Electrical and Electronics Engineers 
integrated product team 

Joint Advanced Distributed Simulation, Albuquerque, New Mexico 
interactive, computer-based simulation of combat operations 
Joint Surveillance Target Attack Radar System 
joint program office 

joint test and evaluation 

joint test force 

kilometers 

local area network 

Live Fly Phase 

light ground station module 

Linked Simulators Phase 

management and integration software 

modeling and simulation 

megabyte 

megabits per second 

military intelligence 

millimeter 

measure of effectiveness 

measure of performance 

multiservice operational test and evaluation 
millisecond 

moving target indicator 

network and engineering 

network coordinator 


software that displays real-time bandwidth use in a rolling bar graph format 


for quick visual reference 
network interface unit 

network time protocol 

operations and control 

Office of the Secretary of Defense 
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OT 
OT&E 
OWS 
PDU 
PM 
POC 
PTP 
PVD 
RPSI 
RSR 
RWS 
SAR 
SCDL 
SE 
sec 

SGI 
SIT 
SM&C 
SME 
Spectrum 
STARS 
STRICOM 
SUT 
SWA 
T&E 
T-1 


TAC 
TAFSM 
TCAC 
TEMPEST 
TEXCOM 
TRAC 
TRADOC 


V&V 
VHF 
VSTARS 
VV&A 
WAN 
WSMR 


XPATCHES 


operational test 

operational test and evaluation 

operator workstation 

protocol data unit 

program manager 

point of contact 

program test plan 

plan view display 

radar processor simulator and integrator 

radar service request 

remote workstation 

synthetic aperture radar 

surveillance control data link 

synthetic environment 

second 

Silicon Graphics, Inc. 

System Integration Test 

system management and control 

subject matter experts 

an instrumentation suite used to measure bandwidth utilization 
surveillance target attack radar system 

U.S. Army Simulation, Training, and Instrumentation Command 
system under test 

Southwest Asia 

test and evaluation , 
digital carrier used to transmit a formatted digital signal at 1.544 megabits 
per second 

target analysis cell 

Tactical Army Fire Support Model 

Test Control and Analysis Center, Albuquerque, New Mexico 
special shielding against electromagnetic radiation 

U.S. Army Test and Experimentation Command 

U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
U.S. Army Training and Doctrine Command 

unmanned aerial vehicle 

user data protocol 

ultra high frequency 

verification and validation 

very high frequency 

Virtual Surveillance Target Attack Radar System 

verification, validation, and accreditation 

wide area network 

White Sands Missile Range 

E-8C synthetic aperture radar simulation developed by Wright Laboratory, 
Dayton, Ohio, and Loral Defense Systems, Goodyear, Arizona 
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